Medical robotic system with coupled control modes

ABSTRACT

In a coupled control mode, the surgeon directly controls movement of an associated slave manipulator with an input device while indirectly controlling movement of one or more non-associated slave manipulators, in response to commanded motion of the directly controlled slave manipulator, to achieve a secondary objective. By automatically performing secondary tasks through coupled control modes, the system&#39;s usability is enhanced by reducing the surgeon&#39;s need to switch to another direct mode to manually achieve the desired secondary objective. Thus, coupled control modes allow the surgeon to better focus on performing medical procedures and to pay less attention to managing the system.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/094,721 (filed Apr. 8, 2016), which is a continuation of U.S. patentapplication Ser. No. 14/095,011 (filed Dec. 3, 2013), now U.S. Pat. No.9,333,042, which is a divisional of U.S. patent application Ser. No.12/780,071 (filed May 14, 2010), now U.S. Pat. No. 8,620,473, which is acontinuation-in-part of U.S. patent application Ser. No. 11/762,200(filed Jun. 13, 2007), now U.S. Pat. No. 7,725,214; each of which isincorporated herein by reference.

U.S. patent application Ser. No. 12/780,071 is also acontinuation-in-part of U.S. patent application Ser. No. 12/489,566(filed Jun. 23, 2009), now U.S. Pat. No. 9,089,256, and U.S. patentapplication Ser. No. 12/780,071 is a continuation-in-part of U.S. patentapplication Ser. No. 12/613,328 (filed Nov. 5, 2009), now U.S. Pat. No.9,084,623, which is a continuation-in-part of U.S. patent applicationSer. No. 12/541,913 (filed Aug. 15, 2009), now U.S. Pat. No. 8,903,546;each of which is incorporated herein by reference.

In addition, this application is related to the following United Statespatent applications, all of which are incorporated by reference:

-   -   U.S. patent application Ser. No. 11/762,217 entitled “Retraction        of tissue for single port entry, robotically assisted medical        procedures” by Mohr;    -   U.S. patent application Ser. No. 11/762,222 entitled “Bracing of        bundled medical devices for single port entry, robotically        assisted medical procedures” by Mohr et al.;    -   U.S. patent application Ser. No. 11/762,231 entitled “Extendable        suction surface for bracing medical devices during robotically        assisted medical procedures” by Schena;    -   U.S. patent application Ser. No. 11/762,236 entitled “Control        system configured to compensate for non-ideal actuator-to-joint        linkage characteristics in a medical robotic system” by Diolaiti        et al.;    -   U.S. patent application Ser. No. 11/762,185 entitled “Surgical        instrument actuation system” by Cooper et al.;    -   U.S. patent application Ser. No. 11/762,172 entitled “Surgical        instrument actuator” by Cooper et al.;    -   U.S. patent application Ser. No. 11/762,165 entitled “Minimally        invasive surgical system” by Larkin et al.;    -   U.S. patent application Ser. No. 11/762,161 entitled “Minimally        invasive surgical instrument advancement” by Larkin et al.;    -   U.S. patent application Ser. No. 11/762,158 entitled “Surgical        instrument control and actuation” by Cooper et al.;    -   U.S. patent application Ser. No. 11/762,154 entitled “Surgical        instrument with parallel motion mechanism” by Cooper;    -   U.S. patent application Ser. No. 11/762,149 entitled “Minimally        invasive surgical apparatus with side exit instruments” by        Larkin;    -   U.S. patent application Ser. No. 11/762,170 entitled “Minimally        invasive surgical apparatus with side exit instruments” by        Larkin;    -   U.S. patent application Ser. No. 11/762,143 entitled “Minimally        invasive surgical instrument system” by Larkin;    -   U.S. patent application Ser. No. 11/762,135 entitled “Side        looking minimally invasive surgery instrument assembly” by        Cooper et al.;    -   U.S. patent application Ser. No. 11/762,132 entitled “Side        looking minimally invasive surgery instrument assembly” by        Cooper et al.;    -   U.S. patent application Ser. No. 11/762,127 entitled “Guide tube        control of minimally invasive surgical instruments” by Larkin et        al.;    -   U.S. patent application Ser. No. 11/762,123 entitled “Minimally        invasive surgery guide tube” by Larkin et al.;    -   U.S. patent application Ser. No. 11/762,120 entitled “Minimally        invasive surgery guide tube” by Larkin et al.;    -   U.S. patent application Ser. No. 11/762,118 entitled “Minimally        invasive surgical retractor system” by Larkin;    -   U.S. patent application Ser. No. 11/762,114 entitled “Minimally        invasive surgical illumination” by Schena et al.;    -   U.S. patent application Ser. No. 11/762,110 entitled “Retrograde        instrument” by Duval et al.;    -   U.S. patent application Ser. No. 11/762,204 entitled “Retrograde        instrument” by Duval et al.;    -   U.S. patent application Ser. No. 11/762,202 entitled “Preventing        instrument/tissue collisions” by Larkin;    -   U.S. patent application Ser. No. 11/762,189 entitled “Minimally        invasive surgery instrument assembly with reduced cross section”        by Larkin et al.;    -   U.S. patent application Ser. No. 11/762,191 entitled “Minimally        invasive surgical system” by Larkin et al.; and    -   U.S. patent application Ser. No. 11/762,196 entitled “Minimally        invasive surgical system” by Duval et al.

BACKGROUND 1. Field of Invention

The present invention generally relates to medical robotic systems andin particular, to a medical robotic system providing coupled controlmodes.

2. Background Art

Minimally invasive surgery is known under various names (e.g.,endoscopy, laparoscopy, arthroscopy, endovascular, keyhole, etc.), oftenspecific to the anatomical area in which work is done. Such surgeryincludes the use of both hand-held andteleoperated/telemanipulated/telepresence (robot assisted/telerobotics)equipment, such as the da Vinci® Surgical System made by IntuitiveSurgical, Inc. of Sunnyvale, Calif. Both diagnostic (e.g., biopsy) andtherapeutic procedures (“medical procedures”) are done. Instruments maybe inserted into a patient percutaneously via surgical incision or vianatural orifice. A new, experimental minimally invasive surgeryvariation is Natural Orifice Transluminal Endoscopic Surgery (NOTES), inwhich instruments enter via a natural orifice (e.g., mouth, nostril, earcanal, anus, vagina, urethra) and continue to a surgical site via atransluminal incision (e.g., in a gastric or colonic wall) within thebody. Although teleoperative surgery using the da Vinci® Surgical Systemprovides great benefits over, for instance, many hand-held procedures,for some patients and for some anatomical areas the da Vinci® SurgicalSystem may be unable to effectively access a surgical site. In addition,further reducing the size and number of incisions generally aids patientrecovery and helps reduce patient trauma and discomfort.

Various slave manipulators are provided in such medical robotic systemsto perform useful functions, such as manipulating instruments to performmedical procedures on a patient, positioning and orienting imagingsystems such as endoscopic imaging devices to capture images of theinstruments' working ends, and delivering the working ends of theinstruments and an image capturing end of the imaging system to a worksite in the patient. The delivery of the working and image capturingends of the instruments and imaging system (“medical devices”) uses oneor more guide tubes and structures that hold and manipulate the guidetube(s). In addition, master manipulators are used as input devices totrack the motion of their operator's hands and to provide appropriatehaptic feedback to the operator indicative of the state of theirassociated slave manipulators. Depending on their respective function,the slave and master manipulators (“robotic manipulators”) may bedesigned with different workspaces and dexterities.

In general, the reachable workspace of a medical device that is beingmanipulated by a slave manipulator is the set of points and orientationsin space that its distal tip (e.g., working or image capturing end) canreach. On the other hand, the dexterous workspace of the medicaldevice's distal tip generally identifies the set of points in space thatcan be reached by primarily changing its orientation (e.g., changing theposition of a wrist joint that orients the distal tip). As explanation,dexterity is a measure of the capability of a robotic manipulator tocontrol the position (in a limited manner) and orientation of theworking end of its associated medical device. Further, it relates thejoint degrees of freedom (i.e. the number of independently actuatedjoints in a kinematic chain of the robotic manipulator/medical device)and the Cartesian/output degrees of freedom that describe theindependent rigid body positions and orientations of the distal tip.While the number of output (slave manipulator) degrees of freedom (DOF)is often at most six, the number of input (master manipulator) jointDOFs varies greatly depending on the master manipulator design

As may be readily appreciated, the dexterous workspace is generally asubset of the reachable workspace. To enable the surgeon to finelycontrol working ends of the instruments, instrument slave manipulatorsare generally designed to optimize their dexterity, even at the expenseof sacrificing their overall reachable workspace. To compensate for suchlimitation, a base manipulator (such as a patient side cart) with alarge reachable workspace may be used to deliver the instrument andimaging system slave manipulators near the entry apertures (e.g.,minimally invasive incisions or natural orifices) in the patient body.Further, when the instruments and imaging system are disposed within acommon guide tube, the guide tube serves as a secondary base sincemovement of the guide tube in this case effectively moves all of theinstruments and the imaging system disposed therein. The instrument andimaging system slave manipulators may then finally deliver the workingand image capturing ends of their respective medical devices to the worksite (e.g., target anatomy) in the patient.

The overall capability of a medical robotic system is achieved by abalance between the workspace and dexterity of all the roboticmanipulators that constitute it. However, the differences in theindividual capabilities of each manipulator have to be clear and wellunderstood by the user in order to effectively utilize the system. It isin general difficult for the user to select which manipulator to controlfrom the console and how to move it in order to achieve a desired“working configuration” of their respective medical devices inside thepatient, with the instruments' working ends having the best possibledexterity and reach, while the capturing end of the imaging system ispositioned in such a way to provide good visualization of the medicalprocedure being performed at the work site without interfering with theinstruments' movements. Hence, it is desirable to provide the systemwith the capability of performing secondary or coupled controlmovements, e.g., for the camera manipulator and the base manipulator(guide tube manipulator and/or manipulator for moving the setup arms andor support for the patient side support system), so as not to distractthe user from performing the medical procedure at the time using thesurgical instruments.

The number of degrees of freedom (DOFs) is the number of independentvariables that uniquely identify the pose/configuration of a system.Since robotic manipulators are kinematic chains that map the (input)joint space into the (output) Cartesian space, the notion of DOF can beexpressed in any of these two spaces. In particular, the set of jointDOFs is the set of joint variables for all the independently controlledjoints. Without loss of generality, joints are mechanisms that provide asingle translational (prismatic joints) or rotational (revolute joints)DOF. Any mechanism that provides more than one DOF motion is considered,from a kinematic modeling perspective, as two or more separate joints.The set of Cartesian DOFs is usually represented by the threetranslational (position) variables (e.g., surge, heave, sway) and by thethree rotational (orientation) variables (e.g. Euler angles orroll/pitch/yaw angles) that describe the position and orientation of anend effector (or tip) frame with respect to a given reference Cartesianframe.

For example, a planar mechanism with an end effector mounted on twoindependent and perpendicular rails has the capability of controllingthe x/y position within the area spanned by the two rails (prismaticDOFs). If the end effector can be rotated around an axis perpendicularto the plane of the rails, then there are then three input DOFs (the tworail positions and the yaw angle) that correspond to three output DOFs(the x/y position and the orientation angle of the end effector).

Although the number of Cartesian DOFs is at most six, a condition inwhich all the translational and orientational variables areindependently controlled, the number of joint DOFs is generally theresult of design choices that involve considerations of the complexityof the mechanism and the task specifications. Accordingly, the number ofjoint DOFs can be more than, equal to, or less than six. Fornon-redundant kinematic chains, the number of independently controlledjoints is equal to the degree of mobility for the end effector frame.For a certain number of prismatic and revolute joint DOFs, the endeffector frame will have an equal number of DOFs (except when insingular configurations) in Cartesian space that will correspond to acombination of translational (x/y/z position) and rotational(roll/pitch/yaw orientation angle) motions.

The distinction between the input and the output DOFs is extremelyimportant in situations with redundant or “defective” kinematic chains(e.g., mechanical manipulators). In particular, “defective” manipulatorshave fewer than six independently controlled joints and therefore do nothave the capability of fully controlling end effector position andorientation. Instead, defective manipulators are limited to controllingonly a subset of the position and orientation variables. On the otherhand, redundant manipulators have more than six joint DOFs. Thus, aredundant manipulator can use more than one joint configuration toestablish a desired 6-DOF end effector pose. In other words, additionaldegrees of freedom can be used to control not just the end effectorposition and orientation but also the “shape” of the manipulator itself.In addition to the kinematic degrees of freedom, mechanisms may haveother DOFs, such as the pivoting lever movement of gripping jaws orscissors blades.

It is also important to consider reference frames for the space in whichDOFs are specified. For example, a single DOF change in joint space(e.g., the joint between two links rotates) may result in a motion thatcombines changes in the Cartesian translational and orientationalvariables of the frame attached to the distal tip of one of the links(the frame at the distal tip both rotates and translates through space).Kinematics describes the process of converting from one measurementspace to another. For example, using joint space measurements todetermine the Cartesian space position and orientation of a referenceframe at the tip of a kinematic chain is “forward” kinematics. UsingCartesian space position and orientation for the reference frame at thetip of a kinematic chain to determine the required joint positions is“inverse” kinematics. If there are any revolute joints, kinematicsinvolves non-linear (trigonometric) functions.

SUMMARY

An object of aspects of the invention is to provide coupled controlmodes in which one or more devices may be directly controlled to achievea primary objective and one or more other devices may be indirectlycontrolled to achieve secondary objectives.

The embodiments of the invention are summarized by the claims thatfollow below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of a robot-assisted minimally invasivetelesurgical system.

FIGS. 2 and 3 are schematic views of a patient side support system in atelesurgical system.

FIG. 4 is a simplified front view of a surgeon's console in atelesurgical system.

FIG. 5 is a schematic view that illustrates aspects of a minimallyinvasive surgical instrument assembly.

FIG. 6 is a schematic view that illustrates aspects of a minimallyinvasive surgical instrument assembly.

FIG. 7 is a schematic side view of a detail of FIG. 6.

FIG. 8 is a diagrammatic perspective view of a surgical instrumentassembly.

FIG. 9 is a schematic view of an interface between a surgical instrumentassembly and an actuator assembly.

FIG. 10 is a perspective view of the proximal segment of a minimallyinvasive surgical instrument.

FIG. 11 is a perspective view of a segment of an actuator assembly thatmates with and actuates the instrument shown in FIG. 10.

FIG. 12 is a diagrammatic perspective view that illustrates mountingminimally invasive surgical instruments and actuator assemblies at theend of a setup arm.

FIG. 13 is another diagrammatic perspective view that illustratesmounting minimally invasive surgical instruments and actuator assembliesat the end of a setup arm.

FIG. 14 is a diagrammatic view of transmission mechanisms associatedwith flexible coaxial guide tubes and instruments.

FIG. 15 is a diagrammatic view of multi-port surgery.

FIG. 16 is another diagrammatic view of multi-port surgery.

FIGS. 17-19 are diagrammatic plan views that illustrate further aspectsof preventing undesired instrument collision with tissue.

FIG. 20 is a diagrammatic view of an image mosaiced output display for asurgeon.

FIG. 21 is a diagrammatic perspective view of an illustrative minimallyinvasive surgical instrument assembly that includes a multi jointedinstrument dedicated to retraction.

FIG. 22 is a block diagram of components used for controlling andselectively associating devices on a patient side support system withinput devices in a telesurgical system.

FIG. 23 is a block diagram of a master/slave control system included inmanipulator controllers in the telesurgical system.

FIGS. 24-25 are block diagrams of a direct “tool following” modearchitecture implemented in the manipulator controllers in thetelesurgical system.

FIGS. 26-27 are block diagrams of a direct “imaging system” modearchitecture implemented in the manipulator controllers in thetelesurgical system.

FIGS. 28-29 are block diagrams of a direct “guide tube” modearchitecture implemented in the manipulator controllers in thetelesurgical system.

FIG. 30 is a diagrammatic view of a centralized motion control systemfor a minimally invasive telesurgical system.

FIG. 31 is a diagrammatic view of a distributed motion control systemfor a minimally invasive telesurgical system.

FIG. 32 is a block diagram of a coupled “tool following” modearchitecture implemented in the manipulator controllers in thetelesurgical system.

FIG. 33 is a block diagram of a coupled “imaging system” modearchitecture implemented in the manipulator controllers in thetelesurgical system.

FIG. 34 is a block diagram of a coupled “guide tube” mode architectureimplemented in the manipulator controllers in the telesurgical system.

FIGS. 35-37 are flow diagrams for an instrument coupled control modeexample.

FIG. 38 is a flow diagram for a guide tube coupled control mode example.

FIG. 39 is a flow diagram for a tool retraction into a fenestrated guidetube for a tool exchange or accessory providing operation in a coupled“tool following” mode example.

FIG. 40 is a flow diagram for an imaging system coupled control modeexample.

DETAILED DESCRIPTION

This description and the accompanying drawings that illustrate aspectsand embodiments of the present invention should not be taken aslimiting—the claims define the protected invention. Various mechanical,compositional, structural, electrical, and operational changes may bemade without departing from the spirit and scope of this description andthe claims. In some instances, well-known circuits, structures, andtechniques have not been shown in detail in order not to obscure theinvention. Like numbers in two or more figures represent the same orsimilar elements.

Further, this description's terminology is not intended to limit theinvention. For example, spatially relative terms—such as “beneath”,“below”, “lower”, “above”, “upper”, “proximal”, “distal”, and thelike—may be used to describe one element's or feature's relationship toanother element or feature as illustrated in the figures. Thesespatially relative terms are intended to encompass different positionsand orientations of the device in use or operation in addition to theposition and orientation shown in the figures. For example, if thedevice in the figures is turned over, elements described as “below” or“beneath” other elements or features would then be “above” or “over” theother elements or features. Thus, the exemplary term “below” canencompass both positions and orientations of above and below. The devicemay be otherwise oriented (rotated 90 degrees or at other orientations),and the spatially relative descriptors used herein interpretedaccordingly. Likewise, descriptions of movement along and around variousaxes includes various special device positions and orientations. Inaddition, the singular forms “a”, “an”, and “the” are intended toinclude the plural forms as well, unless the context indicatesotherwise. And, the terms “comprises”, “comprising”, “includes”, and thelike specify the presence of stated features, steps, operations,elements, and/or components but do not preclude the presence or additionof one or more other features, steps, operations, elements, components,and/or groups. Components described as coupled may be electrically ormechanically directly coupled, or they may be indirectly coupled via oneor more intermediate components.

Telemanipulation and like terms generally refer to an operatormanipulating a master device (e.g., an input kinematic chain) in arelatively natural way (e.g., a natural hand or finger movement),whereupon the master device movements are made into commands that areprocessed and transmitted in real time to a slave device (e.g., anoutput kinematic chain) that reacts nearly instantaneously to thecommands and to environmental forces. Telemanipulation is disclosed inU.S. Pat. No. 6,574,355 (Green), which is incorporated by reference.

To avoid repetition in the figures and the descriptions below of thevarious aspects and illustrative embodiments, it should be understoodthat many features are common to many aspects and embodiments. Omissionof an aspect from a description or figure does not imply that the aspectis missing from embodiments that incorporate that aspect. Instead, theaspect may have been omitted for clarity and to avoid prolixdescription.

Accordingly, several general aspects apply to various descriptionsbelow. For example, at least one surgical end effector is shown ordescribed in various figures. An end effector is the part of theminimally invasive surgical instrument or assembly that performs aspecific surgical function (e.g., forceps/graspers, needle drivers,scissors, electrocautery hooks, staplers, clip appliers/removers, etc.).Many end effectors have a single DOF (e.g., graspers that open andclose). The end effector may be coupled to the surgical instrument bodywith a mechanism that provides one or more additional DOFs, such as“wrist” type mechanisms. Examples of such mechanisms are shown in U.S.Pat. No. 6,371,952 (Madhani et al.) and in U.S. Pat. No. 6,817,974(Cooper et al.), both of which are incorporated by reference, and may beknown as various Intuitive Surgical, Inc. Endowrist® mechanisms as usedon both 8 mm and 5 mm instruments for the da Vinci® Surgical System.Although the surgical instruments described herein generally include endeffectors, it should be understood that in some aspects an end effectormay be omitted. For example, the distal tip of an instrument body shaftmay be used to retract tissue. As another example, suction or irrigationopenings may exist at the distal tip of a body shaft or the wristmechanism. In these aspects, it should be understood that descriptionsof positioning and orienting an end effector include positioning andorienting the tip of a surgical instrument that does not have an endeffector. For example, a description that addresses the reference framefor a tip of an end effector should also be read to include thereference frame of the tip of a surgical instrument that does not havean end effector.

Throughout this description, it should be understood that a mono- orstereoscopic imaging system/image capture component/camera device may beplaced at the distal end of an instrument wherever an end effector isshown or described (the device may be considered a “camera instrument”),or it may be placed near or at the distal end of any guide tube or otherinstrument assembly element. Accordingly, the terms “imaging system” andthe like as used herein should be broadly construed to include bothimage capture components and combinations of image capture componentswith associated circuitry and hardware, within the context of theaspects and embodiments being described. Such endoscopic imaging systems(e.g., optical, infrared, ultrasound, etc.) include systems withdistally positioned image sensing chips and associated circuits thatrelay captured image data via a wired or wireless connection to outsidethe body. Such endoscopic imaging systems also include systems thatrelay images for capture outside the body (e.g., by using rod lenses orfiber optics). In some instruments or instrument assemblies a directview optical system (the endoscopic image is viewed directly at aneyepiece) may be used. An example of a distally positioned semiconductorstereoscopic imaging system is described in U.S. patent application Ser.No. 11/614,661 “Stereoscopic Endoscope” (Shafer et al.), which isincorporated by reference. Well-known endoscopic imaging systemcomponents, such as electrical and fiber optic illumination connections,are omitted or symbolically represented for clarity. Illumination forendoscopic imaging is typically represented in the drawings by a singleillumination port. It should be understood that these depictions areexemplary. The sizes, positions, and numbers of illumination ports mayvary. Illumination ports are typically arranged on multiple sides of theimaging apertures, or completely surrounding the imaging apertures, tominimize deep shadows.

In this description, cannulas are typically used to prevent a surgicalinstrument or guide tube from rubbing on patient tissue. Cannulas may beused for both incisions and natural orifices. For situations in which aninstrument or guide tube does not frequently translate or rotaterelative to its insertion (longitudinal) axis, a cannula may not beused. For situations that require insufflation, the cannula may includea seal to prevent excess insufflation gas leakage past the instrument orguide tube. For example, for thoracic surgery that does not requireinsufflation, the cannula seal may be omitted, and if instruments orguide tube insertion axis movement is minimal, then the cannula itselfmay be omitted. A rigid guide tube may function as a cannula in someconfigurations for instruments that are inserted relative to the guidetube. Cannulas and guide tubes may be, e.g., steel or extruded plastic.Plastic, which is less expensive than steel, may be suitable forone-time use.

Various instances and assemblies of flexible surgical instruments andguide tubes are contemplated as applicable with the present invention.Such flexibility, in this description, is achieved in various ways. Forexample, a segment or an instrument or guide tube may be a continuouslycurving flexible structure, such as one based on a helical wound coil oron tubes with various segments removed (e.g., kerf-type cuts). Or, theflexible part may be made of a series of short, pivotally connectedsegments (“vertebrae”) that provide a snake-like approximation of acontinuously curving structure. Instrument and guide tube structures mayinclude those in U.S. Patent Application Pub. No. US 2004/0138700(Cooper et al.), which is incorporated by reference. For clarity, thefigures and associated descriptions generally show only two segments ofinstruments and guide tubes, termed proximal (closer to the transmissionmechanism; farther from the surgical site) and distal (farther from thetransmission mechanism; closer to the surgical site). It should beunderstood that the instruments and guide tubes may be divided intothree or more segments, each segment being rigid, passively flexible, oractively flexible. Flexing and bending as described for a distalsegment, a proximal segment, or an entire mechanism also apply tointermediate segments that have been omitted for clarity. For instance,an intermediate segment between proximal and distal segments may bend ina simple or compound curve. Flexible segments may be various lengths.Segments with a smaller outside diameter may have a smaller minimumradius of curvature while bending than segments with a larger outsidediameter. For cable-controlled systems, unacceptably high cable frictionor binding limits minimum radius of curvature and the total bend anglewhile bending. The guide tube's (or any joint's) minimum bend radius issuch that it does not kink or otherwise inhibit the smooth motion of theinner surgical instrument's mechanism. Flexible components may be, forexample, up to approximately four feet in length and approximately 0.6inches in diameter. Other lengths and diameters (e.g., shorter, smaller)and the degree of flexibility for a specific mechanism may be determinedby the target anatomy for which the mechanism has been designed.

In some instances only a distal segment of an instrument or guide tubeis flexible, and the proximal segment is rigid. In other instances, theentire segment of the instrument or guide tube that is inside thepatient is flexible. In still other instances, an extreme distal segmentmay be rigid, and one or more other proximal segments are flexible. Theflexible segments may be passive or they may be actively controllable(“steerable”). Such active control may be done using, for example, setsof opposing cables (e.g., one set controlling “pitch” and an orthogonalset controlling “yaw”; three cables can be used to perform similaraction). Other control elements such as small electric or magneticactuators, shape memory alloys, electroactive polymers (“artificialmuscle”), pneumatic or hydraulic bellows or pistons, and the like may beused. In instances in which a segment of an instrument or guide tube isfully or partially inside another guide tube, various combinations ofpassive and active flexibility may exist. For instance, an activelyflexible instrument inside a passively flexible guide tube may exertsufficient lateral force to flex the surrounding guide tube. Similarly,an actively flexible guide tube may flex a passively flexible instrumentinside it. Actively flexible segments of guide tubes and instruments maywork in concert. For both flexible and rigid instruments and guidetubes, control cables placed farther from the center longitudinal axismay provide a mechanical advantage over cables placed nearer to thecenter longitudinal axis, depending on compliance considerations in thevarious designs.

The flexible segment's compliance (stiffness) may vary from being almostcompletely flaccid (small internal frictions exist) to beingsubstantially rigid. In some aspects, the compliance is controllable.For example, a segment or all of a flexible segment of an instrument orguide tube can be made substantially (i.e., effectively but notinfinitely) rigid (the segment is “rigidizable” or “lockable”). Thelockable segment may be locked in a straight, simple curve or in acompound curve shape. Locking may be accomplished by applying tension toone or more cables that run longitudinally along the instrument or guidetube that is sufficient to cause friction to prevent adjacent vertebraefrom moving. The cable or cables may run through a large, central holein each vertebra or may run through smaller holes near the vertebra'souter circumference. Alternatively, the drive element of one or moremotors that move one or more control cables may be soft-locked inposition (e.g., by servocontrol) to hold the cables in position andthereby prevent instrument or guide tube movement, thus locking thevertebrae in place. Keeping a motor drive element in place may be doneto effectively keep other movable instrument and guide tube componentsin place as well. It should be understood that the stiffness underservocontrol, although effective, is generally less than the stiffnessthat may be obtained with braking placed directly on joints, such as thebraking used to keep passive setup joints in place. Cable stiffnessgenerally dominates because it is generally less than servosystem orbraked joint stiffness.

In some situations, the compliance of the flexible segment may becontinuously varied between flaccid and rigid states. For example,locking cable tension can be increased to increase stiffness but withoutlocking the flexible segment in a rigid state. Such intermediatecompliance may allow for telesurgical operation while reducing tissuetrauma that may occur due to movements caused by reactive forces fromthe surgical site. Suitable bend sensors incorporated into the flexiblesegment allow the telesurgical system to determine instrument and/orguide tube position as it bends. U.S. Patent Application Pub. No. US2006/0013523 (Childers et al.), which is incorporated by reference,discloses a fiber optic position shape sensing device and method. U.S.patent application Ser. No. 11/491,384 (Larkin et al.), which isincorporated by reference, discloses fiber optic bend sensors (e.g.,fiber Bragg gratings) used in the control of such segments and flexibledevices.

A surgeon's inputs to control aspects of the minimally invasive surgicalinstrument assemblies, instruments, and end effectors as describedherein are generally done using an intuitive, camera referenced controlinterface. For example, the da Vinci® Surgical System includes aSurgeon's console with such a control interface, which may be modifiedto control aspects described herein. The surgeon manipulates one or moremaster manual input mechanisms having, e.g., 6 DOFs to control the slaveinstrument assembly and instrument. The input mechanisms include afinger-operated grasper to control one or more end effector DOFs (e.g.,closing grasping jaws). Intuitive control is provided by orienting therelative positions of the end effectors and the endoscopic imagingsystem with the positions of the surgeon's input mechanisms and imageoutput display. This orientation allows the surgeon to manipulate theinput mechanisms and end effector controls as if viewing the surgicalwork site in substantially true presence. This teleoperation truepresence means that the surgeon views an image from a perspective thatappears to be that of an operator directly viewing and working at thesurgical site. U.S. Pat. No. 6,671,581 (Niemeyer et al.), which isincorporated by reference, contains further information on camerareferenced control in a minimally invasive surgical apparatus.

FIG. 1 is a schematic view that illustrates aspects of a robot-assisted(telemanipulative) minimally invasive surgical system 2100 in whichinstruments are inserted in a patient through a single entry aperturethrough a guide tube. This system's general architecture is similar tothe architecture of other such systems such as Intuitive Surgical,Inc.'s da Vinci® Surgical System and the Zeus® Surgical System. Thethree main components are a surgeon's console 2102, a patient sidesupport system 2104, and a video system 2106, all interconnected 2108 bywired or wireless connections as shown.

As shown in FIG. 4, the surgeon's console 2102 includes, e.g.,hand-operable, multiple DOF mechanical input (“master”) devices 203, 204and foot pedals 215, 217 that allow the surgeon to manipulate thesurgical instruments, guide tubes, and imaging system (“slave”) devicesas described herein. These input devices may in some aspects providehaptic feedback from the instruments and instrument assembly componentsto the surgeon. Buttons 205, 207 are provided on the hand-operable inputdevices 203, 204 for switching functions as described herein or forother operational purposes. Console 2102 also includes a stereoscopicvideo output display 201 positioned such that images on the display aregenerally focused at a distance that corresponds to the surgeon's handsworking behind/below the display screen. A processor 220 incommunication with other components of the console via bus 210 performsvarious functions in the system 2100. One important function that itperforms is to implement the various controllers described herein totranslate and transfer the mechanical motion of input devices throughcontrol signals so that the Surgeon can effectively manipulate andotherwise move devices, such as the surgical instruments, an imagingsystem, and one or more guide tubes, that are selectively associatedwith the input devices at the time. Although described as a processor,it is to be appreciated that the processor 220 may be implemented inpractice by any combination of hardware, software and firmware. Also,its functions as described herein may be performed by one unit ordivided up among different components, each of which may be implementedin turn by any combination of hardware, software and firmware. Further,although being shown as part of or being physically adjacent to theconsole 2102, the processor 220 may also comprise a number of subunitsdistributed throughout the system. These aspects are discussed morefully in U.S. Pat. No. 6,671,581, which is incorporated by referenceabove.

Referring back to FIG. 1, the patient side support system 2104 includesa floor-mounted structure 2110, or alternately a ceiling mountedstructure 2112 as shown by the alternate lines. The structure 2110 maybe movable or fixed (e.g., to the floor, ceiling, or other equipmentsuch as an operating table). In one embodiment a set-up arm assembly2114 is a modified da Vinci® Surgical System arm assembly. The armassembly 2114 includes two illustrative passive rotational setup joints2114 a,2114 b, which allow manual positioning of the coupled links whentheir brakes are released. A passive prismatic setup joint (not shown)between the arm assembly and the structure 2110 may be used to allow forlarge vertical adjustments. In addition, a guide tube manipulator 2116includes illustrative active roll joint 2116 a and active yaw joint 2116b. Joints 2116 c and 2116 d act as a parallel mechanism so that a guidetube (of a surgical instrument assembly) held by a platform 2118 movesaround remote center 2120 at an entry port, such as patient 1222'sumbilicus. In one embodiment, an active prismatic joint 2124 is used toinsert and withdraw the guide tube. One or more surgical instruments andan endoscopic imaging system are independently mounted to platform 2118.The various setup and active joints allow the manipulators to move theguide tube, instruments, and imaging system when patient 2122 is placedin various positions on movable table 2126.

FIGS. 2 and 3 are schematic side and front elevation views of anotherillustrative embodiment of a patient side support system. Support 2150is fixed (e.g., floor or ceiling mounted). Link 2152 is coupled tosupport 2150 at passive rotational setup joint 2154. As shown, joint2154's rotational axis is aligned with remote center point 2156, whichis generally the position at which a guide tube (of a surgicalinstrument assembly; not shown) enters the patient (e.g., at theumbilicus for abdominal surgery). Link 2158 is coupled to link 2152 atrotational joint 2160. Link 2162 is coupled to link 2158 at rotationaljoint 2164. Link 2166 is coupled to link 2162 at rotational joint 2168.The guide tube is mounted to slide through the end 2166 a of link 2166.Platform 2170 is supported and coupled to link 2166 by a prismatic joint2172 and a rotational joint 2174. Prismatic joint 2172 inserts andwithdraws the guide tube as it slides along link 2166. Joint 2174includes a bearing assembly that holds a “C” shaped ring cantilever. Asthe “C” ring slides through the bearing it rotates around a center pointinside the “C”, thereby rolling the guide tube. The opening in the “C”allows guide tubes to be mounted or exchanged without moving overlyingmanipulators. Platform 2170 supports multiple manipulators 2176 forsurgical instruments and an imaging system, as described below.

These illustrative robotic arm assemblies are used, for example, forinstrument assemblies that include a rigid guide tube and are operatedto move with reference to a remote center. Certain setup and activejoints in the manipulator arm may be omitted if motion around a remotecenter is not required. It should be understood that manipulator armsmay include various combinations of links, passive, and active joints(redundant DOFs may be provided) to achieve a necessary range of posesfor surgery.

Referring again to FIG. 1, video system 2106 performs image processingfunctions for, e.g., captured endoscopic imaging data of the surgicalsite and/or preoperative or real time image data from other imagingsystems external to the patient. Video system 2106 outputs processedimage data (e.g., images of the surgical site, as well as relevantcontrol and patient information) to the surgeon at the surgeon's console2102. In some aspects the processed image data is output to an optionalexternal monitor visible to other operating room personnel or to one ormore locations remote from the operating room (e.g., a surgeon atanother location may monitor the video; live feed video may be used fortraining; etc.).

As an example of an instrument assembly, FIG. 5 is a schematic view thatillustrates aspects of a minimally invasive surgical instrument assembly1600. Two surgical instruments 1602 a,1602 b extend through channels1604 a,1604 b that extend longitudinally through rigid guide tube 1606.In some aspects guide tube 1606 is straight and in others it is curvedto accommodate a particular insertion port (the instruments aresimilarly curved to facilitate insertion). Guide tube 1606 may havevarious cross-sectional shapes (e.g., circular, oval, rounded polygon),and various numbers of surgical instruments and channels may be used.Some optional working channels may be used to provide supportingsurgical functions such as irrigation and suction. In some aspects anendoscopic imaging system (e.g., mono- or stereoscopic image capture ordirect view) is at guide tube 1606's distal end 1610. In one aspectguide tube 1606 is inserted into a patient via an incision (e.g.,approximately 2.0 cm at the umbilicus) or natural orifice, either withor without the use of a cannula 1612 or similar guiding structure. Insome aspects guide tube 1606 may rotate within cannula 1612.

Surgical instruments 1602 a and 1602 b function in a like manner, andmany instrument functions (body roll, wrist operation, end effectoroperation, etc.) are similar to the surgical instruments used in the daVinci® Surgical System (both 8 mm and 5 mm instrument body diameters).In other aspects the instruments may function differently and/or havecapabilities not embodied in da Vinci® Surgical System instruments(e.g., one instrument may be straight, one instrument may be jointed,one instrument may be flexible, etc.). In the present example,instrument 1602 a includes a transmission portion (not shown) at itsproximal end, an elongated instrument body 1614, one of various surgicalend effectors 1616, and a snake-like, two degree of freedom wristmechanism 1618 that couples end effector 1616 to instrument body 1614.As in the da Vinci® Surgical Systems, in some aspects the transmissionportion includes disks that interface with electrical actuators (e.g.,servomotors) permanently mounted on a support arm so that instrumentsmay easily be changed. Other linkages such as matching gimbal plates andlevers may be used to transfer actuating forces at the mechanicalinterface. Mechanical mechanisms (e.g., gears, levers, gimbals) in thetransmission portion transfer the actuating forces from the disks tocables, wires, and/or cable, wire, and hypotube combinations that runthrough one or more channels in instrument body 1614 (which may includeone or more articulated segments) to control wrist 1618 and end effector1616 movement. In some aspects, one or more disks and associatedmechanisms transfer actuating forces that roll instrument body 1614around its longitudinal axis 1619 as shown. In some aspects theactuators for a particular instrument are themselves mounted on a singlelinear actuator that moves instrument body 1614 longitudinally as shownwithin channel 1604 a. The main segment of instrument body 1614 is asubstantially rigid single tube, although in some aspects it may beslightly resiliently flexible. This small flexibility allows a proximalbody segment 1620 proximal of guide tube 1606 (i.e., outside thepatient) be slightly flexed so that several instrument bodies can bespaced more closely within guide tube 1606 than their individualtransmission segment housings would otherwise allow, like several cutflowers of equal length being placed in a small-necked vase. Thisflexing is minimal (e.g., less than or equal to about a 5-degree bendangle in one embodiment) and does not induce significant frictionbecause the bend angle for the control cables and hypotubes inside theinstrument body is small.

Instruments 1602 a and 1602 b each include a proximal body segment thatextends through the guide tube and at least one distal body segment thatis positioned beyond the guide tube's distal end. For example,instrument 1602 a includes proximal body segment 1620 that extendsthrough guide tube 1606, a distal body segment 1622 that is coupled toproximal body segment 1620 at a joint 1624, a wrist mechanism 1626 thatis coupled to distal body segment 1622 at another joint 1628 (thecoupling may include another, short distal body segment), and an endeffector 1630. In some aspects the distal body segment 1622 and joints1624 and 1628 function as a parallel motion mechanism 1632 in which theposition of a reference frame at the distal end of the mechanism may bechanged with respect to a reference frame at the proximal end of themechanism without changing the orientation of the distal referenceframe.

FIG. 6 is a schematic view that illustrates aspects of another minimallyinvasive surgical instrument assembly 1700. Surgical instrument assembly1700 is similar to instrument assembly 1600 in that surgical instruments1702 a,1702 b function similarly to instruments 1602 a,1602 b asdescribed above, but instead of a fixed endoscopic imaging system at theend of the guide tube, assembly 1700 has an independently operatingendoscopic imaging system 1704.

In one aspect, imaging system 1704 is mechanically similar to surgicalinstruments 1602 as described above. Summarizing these aspects as shownin FIG. 6, optical system 1704 includes a substantially rigid elongatetubular proximal body segment 1706 that extends through guide tube 1708,and at proximal body segment 1706's distal end there is coupled a 1 or 2DOF parallel motion mechanism 1712 that is similar to parallel motionmechanism 1622. Parallel motion mechanism 1712 includes a first joint1714, an intermediate distal body segment 1716, and a second joint 1718.A wrist mechanism or other active joint (e.g., one DOF to allow changingpitch angle; two DOFs to allow changing pitch and yaw angles) 1720couples an image capture component 1722 to second joint 1718.Alternatively, joint 1714 is an independently controllable one or twoDOF joint (pitch/yaw), joint 1718 is another independently controllableone or two DOF joint (e.g., pitch/yaw), and image capture component 1722is coupled directly at the distal end of the joint 1718 mechanism. Anexample of a suitable stereoscopic image capture component is shown inU.S. patent application Ser. No. 11/614,661, incorporated by referenceabove. In some aspects imaging system 1704 moves longitudinally (surges)inside guide tube 1708. Control of imaging system 1704 is furtherdescribed in concurrently filed U.S. patent application Ser. No.11/762,236, incorporated by reference above. In some aspects, roll maybe undesirable because of a need to preserve a particular field of vieworientation. Having heave (up/down), sway (side-to-side), surge(retraction/insertion), yaw, and pitch DOFs allows the image capturecomponent to be moved to various positions while preserving a particularcamera reference for assembly 1700 and viewing alignment for thesurgeon.

FIG. 7 is, for illustrative purposes only, a side view schematic to FIG.6's plan view schematic. FIG. 7 shows that parallel motion mechanism1712 moves image capture component 1722 away from surgical instrumentassembly 1700's longitudinal centerline. This displacement provides animproved view of surgical site 1724 because some or all of theinstrument body distal segment ends are not present in the image outputto the surgeon as would occur in, e.g., instrument assembly 1600 (FIG.5). The pitch of parallel motion mechanism 1712 and of image capturecomponent 1722 is controllable, as illustrated by the arrows.

FIG. 8 is a diagrammatic perspective view that illustrates an embodimentof surgical instrument assembly 1700. As shown, two independentlyteleoperated surgical instruments 1740 a,1740 b (each instrument isassociated with a separate master—e.g. one left hand master for the leftinstrument and one right hand master for the right instrument) runthrough and emerge at the distal end of a rigid guide tube 1742. Eachinstrument 1740 a,1740 b is a 6 DOF instrument, as described above, andincludes a parallel motion mechanism 1744 a,1744 b, as described above,with wrists 1746 a,1746 b and end effectors 1748 a,1748 b attached. Inaddition, an independently teleoperated endoscopic imaging system 1750runs through and emerges at the distal end of guide tube 1742. In someaspects imaging system 1750 also includes a parallel motion mechanism1752, a pitch-only wrist mechanism 1754 at the distal end of theparallel motion mechanism 1752 (the mechanism may have either one or twoDOFs in joint space), and a stereoscopic endoscopic image capturecomponent 1756 coupled to wrist mechanism 1754. In other aspects, wristmechanism 1754 may include a yaw DOF. In yet another aspect, theproximal and distal joints in imaging system 1750 are independentlycontrolled. In an illustrative use, parallel motion mechanism 1752heaves and sways image capture component 1756 up and to the side, andwrist mechanism 1754 orients image capture component 1756 to place thecenter of the field of view between the instrument tips if theinstruments are working to the side of the guide tube's extendedcenterline. In another illustrative use, the distal body segment ofimaging system is independently pitched up (in some aspects alsoindependently yawed), and image capture component 1756 is independentlypitched down (in some aspects also independently yawed). As discussedabove and below, imaging system 1750 may be moved to various places toretract tissue.

Also shown is an auxiliary channel 1760, through which, e.g.,irrigation, suction, or other surgical items may be introduced orwithdrawn. In some aspects, one or more small, steerable devices may beinserted via auxiliary channel 1760 to spray a cleaning fluid (e.g.,pressurized water, gas) and/or a drying agent (e.g., pressurized air orinsufflation gas) on the imaging system's windows to clean them. Inanother aspect, such a cleaning wand may be a passive device thatattaches to the camera before insertion. In yet another aspect, the endof the wand is automatically hooked to the image capture component asthe image capture component emerges from the guide tube's distal end. Aspring gently pulls on the cleaning wand so that it tends to retractinto the guide tube as the imaging system is withdrawn from the guidetube.

FIG. 7 further illustrates that as image capture component 1722 is movedaway from assembly 1700's centerline it may press against and move anoverlying tissue structure surface 1726, thereby retracting the tissuestructure from the surgical site as shown. The use of imaging system1704 to retract tissue is illustrative of using other surgicalinstruments, or a device specifically designed for the task, to retracttissue. Such “tent-pole” type retraction may be performed by any of thevarious movable components described herein, such as the distal end exitor side exit flexible devices and the parallel motion mechanisms on therigid body component devices, as well as other devices discussed below(e.g., with reference to FIG. 21).

FIG. 9 is a schematic view that illustrates aspects of an interfacebetween surgical instrument assembly 2302, which represents flexible andrigid mechanisms as variously described herein, and an illustrativeactuator assembly 2304. For the purposes of this example, instrumentassembly 2302 includes surgical instrument 2306, primary guide tube 2308that surrounds instrument 2306, and secondary guide tube 2310 thatsurrounds primary guide tube 2308.

As shown in FIG. 9, a transmission mechanism is positioned at theproximal ends of each instrument or guide tube: transmission mechanism2306 a for instrument 2306, transmission mechanism 2308 a for primaryguide tube 2308, and transmission mechanism 2310 a for secondary guidetube 2310. Each transmission mechanism is mechanically and removablycoupled to an associated actuator mechanism: transmission mechanism 2306a to actuator mechanism 2312, transmission mechanism 2308 a to actuatormechanism 2314, transmission mechanism 2310 a to actuator mechanism2316. In one aspect, mating disks are used as in the da Vinci® SurgicalSystem instrument interface, as shown in more detail below. In anotheraspect, mating gimbal plates and levers are used. Various mechanicalcomponents (e.g., gears, levers, cables, pulleys, cable guides, gimbals,etc.) in the transmission mechanisms are used to transfer the mechanicalforce from the interface to the controlled element. Each actuatormechanism includes at least one actuator (e.g., servomotor (brushed orbrushless)) that controls movement at the distal end of the associatedinstrument or guide tube. For example, actuator 2312 a is an electricservomotor that controls surgical instrument 2306's end effector 2306 bgrip DOF. An instrument (including a guide probe as described herein) orguide tube (or, collectively, the instrument assembly) may be decoupledfrom the associated actuator mechanism(s) and slid out as shown. It maythen be replaced by another instrument or guide tube. In addition to themechanical interface there is an electronic interface between eachtransmission mechanism and actuator mechanism. This electronic interfaceallows data (e.g., instrument/guide tube type) to be transferred.

In some instances one or more DOFs may be manually actuated. Forinstance, surgical instrument 2306 may be a passively flexiblelaparoscopic instrument with a hand-actuated end effector grip DOF, andguide tube 2308 may be actively steerable to provide wrist motion asdescribed above. In this example, the surgeon servocontrols the guidetube DOFs and an assistant hand controls the instrument grip DOF.

In addition to the actuators that control the instrument and/or guidetube elements, each actuator assembly may also include an actuatorcomponent (e.g., motor-driven cable, lead screw, pinion gear, etc.;linear motor; and the like) that provides motion along instrumentassembly 2302's longitudinal axis (surge). As shown in the FIG. 9example, actuator mechanism 2312 includes linear actuator 2312 b,actuator mechanism 2314 includes linear actuator 2314 b, and actuatormechanism 2316 includes linear actuator 2316 b, so that instrument 2306,primary guide tube 2308, and secondary guide tube 2310 can each beindependently coaxially moved. As further shown in FIG. 9, actuatorassembly 2316 is mounted to setup arm 2318, either passively or activelyas described above. In active mounting architectures, the activemounting may be used to control one or more component DOFs (e.g.,insertion of a rigid guide tube).

Control signals from control system 2320 control the various servomotoractuators in actuator assembly 2304. The control signals are, e.g.,associated with the surgeon's master inputs at input/output system 2322to move instrument assembly 2302's mechanical slave components. In turn,various feedback signals from sensors in actuator assembly 2304, and/orinstrument assembly 2302, and/or other components are passed to controlsystem 2320. Such feedback signals may be pose information, as indicatedby servomotor position or other position, orientation, and forceinformation, such as may be obtained with the use of fiber Bragggrating-based sensors. Feedback signals may also include force sensinginformation, such as tissue reactive forces, to be, e.g., visually orhaptically output to the surgeon at input/output system 2322.

Image data from an endoscopic imaging system associated with instrumentassembly 2302 are passed to image processing system 2324. Such imagedata may include, e.g., stereoscopic image data to be processed andoutput to the surgeon via input/output system 2322 as shown. Imageprocessing may also be used to determine instrument position, which isinput to the control system as a form of distal position feedbacksensor. In addition, an optional sensing system 2326 positioned outsideand near the patient may sense position or other data associated withinstrument assembly 2302. Sensing system 2326 may be static or may becontrolled by control system 2320 (the actuators are not shown, and maybe similar to those depicted or to known mechanical servo components),and it may include one or more actual sensors positioned near thepatient. Position information (e.g., from one or more wirelesstransmitters, RFID chips, etc.) and other data from sensing system 2326may be routed to control system 2320. If such position information orother data is to be visually output to the surgeon, control system 2320passes it in either raw or processed form to image processing system2324 for integration with the surgeon's output display at input/outputsystem 2322. Further, any image data, such as fluoroscopic or otherreal-time imaging (ultrasound, X-ray, Mill, and the like), from sensingsystem 2326 are sent to image processing system 2324 for integrationwith the surgeon's display. And, real-time images from sensing system2326 may be integrated with preoperative images accessed by imageprocessing system 2324 for integration with the surgeon's display. Inthis way, for instance, preoperative images of certain tissue (e.g.,brain tissue structures) are received from a data storage location 2328,may be enhanced for better visibility, the preoperative images areregistered with other tissue landmarks in real time images, and thecombined preoperative and real time images are used along with positioninformation from instrument and actuator assemblies 2302,2304 and/orsensing system 2326 to present an output display that assists thesurgeon to maneuver instrument assembly 2302 towards a surgical sitewithout damaging intermediate tissue structures.

FIG. 10 is a perspective view of the proximal portion of a minimallyinvasive surgical instrument 2402. As shown in FIG. 10, instrument 2402includes a transmission mechanism 2404 coupled to the proximal end of aninstrument body tube 2406. Components at body tube 2406's distal end2408 are omitted for clarity and may include, e.g., the 2 DOF parallelmotion mechanism, wrist, and end effector combination as describedabove; joints and an endoscopic imaging system as described above; etc.In the illustrative embodiment shown, transmission mechanism 2404includes six interface disks 2410. One or more disks 2410 are associatedwith a DOF for instrument 240. For instance, one disk may be associatedwith instrument body roll DOF, and a second disk may be associated withend effector grip DOF. As shown, in one instance the disks are arrangedin a hexagonal lattice for compactness—in this case six disks in atriangular shape. Other lattice patterns or more arbitrary arrangementsmay be used. Mechanical components (e.g., gears, levers, gimbals,cables, etc.) inside transmission mechanism 2404 transmit roll torqueson disks 2410 to e.g., body tube 2406 (for roll) and to componentscoupled to distal end mechanisms. Cables and/or cable and hypotubecombinations that control distal end DOFs run through body tube 2406. Inone instance the body tube is approximately 7 mm in diameter, and inanother instance it is approximately 5 mm in diameter. Raised pins 2412,spaced eccentrically, provide proper disk 2410 orientation when matedwith an associated actuator disk. One or more electronic interfaceconnectors 2414 provide an electronic interface between instrument 2402and its associated actuator mechanism. In some instances instrument 2402may pass information stored in a semiconductor memory integrated circuitto the control system via its associated actuator mechanism. Such passedinformation may include instrument type identification, number ofinstrument uses, and the like. In some instances the control system mayupdate the stored information (e.g., to record number of uses todetermine routine maintenance scheduling or to prevent using aninstrument after a prescribed number of times). U.S. Pat. No. 6,866,671(Tierney et al.), which discusses storing information on instruments, isincorporated by reference. The electronic interface may also includepower for, e.g., an electrocautery end effector. Alternately, such apower connection may be positioned elsewhere on instrument 2402 (e.g.,on transmission mechanism 2404's housing). Other connectors for, e.g.,optical fiber lasers, optical fiber distal bend or force sensors,irrigation, suction, etc. may be included. As shown, transmissionmechanism 2404's housing is roughly wedge- or pie-shaped to allow it tobe closely positioned to similar housings, as illustrated below.

FIG. 11 is a perspective view of a portion of an actuator assembly 2420that mates with and actuates components in surgical instrument 2402.Actuator disks 2422 are arranged to mate with interface disks 2410.Holes 2424 in disks 2422 are aligned to receive pins 2412 in only asingle 360-degree orientation. Each disk 2422 is turned by an associatedrotating servomotor actuator 2426, which receives servocontrol inputs asdescribed above. A roughly wedge-shaped mounting bracket 2428, shaped tocorrespond to instrument 2402's transmission mechanism housing, supportsthe disks 2422, servomotor actuators 2426, and an electronic interface2430 that mates with instrument 2402's interface connectors 2414. In oneinstance instrument 2402 is held against actuator assembly 2420 byspring clips (not shown) to allow easy removal. As shown in FIG. 11, aportion 2432 of actuator assembly housing 2428 is truncated to allowinstrument body tube 2406 to pass by. Alternatively, a hole may beplaced in the actuator assembly to allow the body tube to pass through.Sterilized spacers (reusable or disposable; usually plastic) may be usedto separate the actuator assembly and the instrument's transmissionmechanism to maintain a sterile surgical field. A sterile thin plasticsheet or “drape” (e.g., 0.002-inch thick polyethylene) is used to coverportions of the actuator assembly not covered by the spacer, as well asto cover portions of the manipulator arm. U.S. Pat. No. 6,866,671,incorporated by reference above, discusses such spacers and drapes.

FIG. 12 is a diagrammatic perspective view that illustrates aspects ofmounting minimally invasive surgical instruments and their associatedactuator assemblies at the end of a setup/manipulator arm. As shown inFIG. 12, surgical instrument 2502 a is mounted on actuator assembly2504, so that the transmission mechanism mates with the actuatorassembly (optional spacer/drape is not shown) as described above.Instrument 2502 a's body tube 2506 extends past actuator assembly 2504and enters a port in rigid guide tube 2508. As depicted, body tube 2506,although substantially rigid, is bent slightly between the transmissionmechanism housing and the guide tube as discussed above with referenceto FIG. 5. This bending allows the instrument body tube bores in theentry guide to be spaced closer than the size of their transmissionmechanisms would otherwise allow. Since the bend angle in the rigidinstrument body tube is less than the bend angle for a flexible (e.g.,flaccid) instrument body, cables can be stiffer than in a flexible body.High cable stiffness is important because of the number of distal DOFsbeing controlled in the instrument. Also, the rigid instrument body iseasier to insert into a guide tube than a flexible body. In oneembodiment the bending is resilient so that the body tube assumes itsstraight shape when the instrument is withdrawn from the guide tube (thebody tube may be formed with a permanent bend, which would preventinstrument body roll). Actuator assembly 2504 is mounted to a linearactuator 2510 (e.g. a servocontrolled lead screw and nut or a ball screwand nut assembly) that controls body tube 2506's insertion within guidetube 2508. The second instrument 2502 b is mounted with similarmechanisms as shown. In addition, an imaging system (not shown) may besimilarly mounted.

FIG. 12 further shows that guide tube 2508 is removably mounted tosupport platform 2512. This mounting may be, for example, similar to themounting used to hold a cannula on a da Vinci® Surgical Systemmanipulator arm. Removable and replaceable guide tubes allow differentguide tubes that are designed for use with different procedures to beused with the same telemanipulative system (e.g., guide tubes withdifferent cross-sectional shapes or various numbers and shapes ofworking and auxiliary channels). In turn, actuator platform 2512 ismounted to robot manipulator arm 2514 (e.g., 4 DOF) using one or moreadditional actuator mechanisms (e.g., for pitch, yaw, roll, insertion).In turn, manipulator arm 2514 may be mounted to a passive setup arm, asdescribed above with reference to FIG. 1.

FIG. 13 is a diagrammatic perspective view that illustrates aspectsshown in FIG. 12 from a different angle and with reference to a patient.In FIG. 13, arm 2514 and platform 2512 are positioned so that guide tube2508 enters the patient's abdomen at the umbilicus. This entry isillustrative of various natural orifice and incision entries, includingpercutaneous and transluminal (e.g., transgastric, transcolonic,transrectal, transvaginal, transrectouterine (Douglas pouch), etc.)incisions. FIG. 13 also illustrates how the linear actuators for eachinstrument/imaging system operate independently by showing imagingsystem 2518 inserted and instruments 2502 a,2502 b withdrawn. Theseaspects may apply to other surgical instrument assemblies describedherein (e.g., flexible guide tubes with end- or side-exit ports, sideworking tools, etc.). It can be seen that in some instances themanipulator arm moves to rotate guide tube 2508 around a remote center2520 at the entry port into a patient. If intermediate tissue restrictsmovement around a remote center, however, the arm can maintain guidetube 2508 in position.

FIG. 14 is a diagrammatic view that illustrates aspects of transmissionmechanisms associated with flexible coaxial guide tubes and instruments.FIG. 14 shows primary guide tube 2702 running coaxially through andexiting the distal end of secondary guide tube 2704. Likewise, secondaryguide tube 2704 runs coaxially through and exits the distal end oftertiary guide tube 2706. Transmission and actuator mechanism 2708 isassociated with tertiary guide tube 2706. Transmission and actuatormechanism 2710 is associated with secondary guide tube 2704, and aproximal segment of guide tube 2704 extends through (alternatively,adjacent to) transmission and actuator mechanism 2710 before enteringtertiary guide tube 2706. Likewise, transmission and actuator mechanism2712 is associated with primary guide tube 2702, and a proximal segmentof guide tube 2702 extends through (alternatively, adjacent to)transmission and actuator mechanisms 2708,2710 before entering secondaryand tertiary guide tubes 2704,2706. Transmission mechanisms forinstruments and an imaging system (not shown) running through andexiting the distal ends of channels 2714 in primary guide tube 2702 maybe similarly stacked generally along the instrument assembly'slongitudinal axis, or they may be arranged around guide tube 2702'sextended longitudinal axis at its proximal end as described above. Or,the controller positions may be combined side-by-side and stacked, suchas for a side-exit assembly in which transmission mechanisms for theside-exiting components are positioned side-by-side, and both arestacked behind the guide tube transmission mechanism. Intermediate exitassemblies may be similarly configured. Instrument and/or imaging systemactuators and controls may also be combined within the same housing asan actuator and transmission mechanism for a guide tube.

In many aspects the devices described herein are used as single-portdevices—all components necessary to complete a surgical procedure enterthe body via a single entry port. In some aspects, however, multipledevices and ports may be used. FIG. 15 is a diagrammatic view thatillustrates multi-port aspects as three surgical instrument assembliesenter the body at three different ports. Instrument assembly 2802includes a primary guide tube, a secondary guide tube, and twoinstruments, along with associated transmission and actuator mechanisms,as described above. In this illustrative example, instrument assembly2804 includes a primary guide tube, a secondary guide tube, and a singleinstrument, along with associated transmission and actuator mechanisms,as described above. Imaging system assembly 2806 includes a guide tubeand an imaging system, along with associated transmission and actuatormechanisms, as described above. Each of these mechanisms 2802,2804,2806enters the body 2808 via a separate, unique port as shown. The devicesshown are illustrative of the various rigid and flexible aspectsdescribed herein.

FIG. 16 is another diagrammatic view that illustrates multi-portaspects. FIG. 16 shows three illustrative instruments or assemblies 2810entering different natural orifices (nostrils, mouth) and thencontinuing via a single body lumen (throat) to reach a surgical site.

FIGS. 17-19 are diagrammatic plan views that illustrate aspects ofpreventing undesired instrument collision with tissue. Instruments maycollide with patient tissue outside of an imaging system's field of viewin spaces confined by patient anatomy (e.g., laryngeal surgery). Suchcollisions may damage tissue. For multi-DOF surgical instruments, someDOFs may be inside the field of view while other, more proximal DOFs maybe outside the field of view. Consequently, a surgeon may be unawarethat tissue damage is occurring as these proximal DOFs move. As shown inFIG. 17, for example, an endoscopic imaging system 2920 extends from theend of guide tube 2922. The left side working instrument 2924 a isplaced so that all DOFs are within imaging system 2920's field of view2926 (bounded by the dashed lines). The right side working instrument2924 b, however, has proximal DOFs (an illustrative parallel motionmechanism as described above and wrist are shown) that are outside fieldof view 2926, even though instrument 2924 b's end effector is withinfield of view 2926. This instrument position is illustrative of taskssuch as tying sutures.

In one aspect, field of view boundaries can be determined when thecamera is manufactured so that the boundaries are known in relation tothe camera head (image capture component). The boundary information isthen stored in a nonvolatile memory associated with the imaging systemthat incorporates the camera head. Consequently, the control system canuse the imaging system instrument's kinematic and joint positioninformation to locate the camera head relative to the workinginstruments, and therefore the control system can determine the field ofview boundaries relative to the working instruments. Instruments arethen controlled to work within the boundaries.

In another aspect for stereoscopic imaging systems, field of viewboundaries can be determined relative to the instruments by usingmachine vision algorithms to identify the instruments and theirpositions in the field of view. This “tool tracking” subject isdisclosed in U.S. Patent Application Publication No. US 2006/0258938 A1(Hoffman et al.), which is incorporated by reference.

As shown in FIG. 18, imaging system 2920 is placed so that the camerahead is just at the distal end of guide tube 2922. Instruments 2924 aand 2924 b are extended from the distal end of the guide tube and withinimaging system 2920's field of view. An “Allowable Volume” is defined tobe coincident with the field of view boundaries. The control systemprevents any part of instruments 2924 a and 2924 b from moving outsidethe Allowable Volume. Since the surgeon can see all distal moving partsof instruments 2924 a and 2924 b, the surgeon then moves the instrumentswithout colliding with surrounding tissue. The instrument movements arerecorded, and an “Instrument Volume” 2928 (bounded by the dotted lines),which is bounded by the farthest movements of the instruments, isdetermined. The Instrument Volume is a convex volume within whichinstruments may be moved without colliding with tissue.

Next, imaging system 2920 is inserted as shown in FIG. 19. As a result,field of view 2926 is also inserted, and parts of instruments 2924a,2924 b are outside of the inserted field of view 2926. A new AllowableVolume is determined to be the newly inserted field of view plus thepreviously determined Instrument Volume that is outside of the field ofview. Therefore, the control system will allow the surgeon to move aninstrument anywhere within the new Allowable Volume. The process may berepeated for further field of view insertions or for guide tube 2922movements. This scheme allows a surgeon to define the allowableinstrument range of motion in real time without requiring a tissuemodel. The surgeon is only required to trace the boundaries of theinstrument range of motion inside the field of view, and the controlsystem will record this information as the field of view is changed.

Another way to prevent unwanted instrument/tissue collision is by usingimage mosaicing. FIG. 20 is a diagrammatic view of a display (e.g.,stereoscopic) that a surgeon sees during a surgical procedure. As shownin FIG. 20, the image from the new, more inserted field of view 2940(bounded by the dashed lines) is registered and mosaiced with the imagefrom the old, more withdrawn field of view 2942. Image mosaicing isknown (see e.g., U.S. Pat. No. 4,673,988 (Jansson et al.) and U.S. Pat.No. 5,999,662 (Burt et al.), which are incorporated by reference) andhas been applied to medical equipment (see e.g., U.S. Pat. No. 7,194,118(Harris et al.), which is incorporated by reference). As a result, thesurgeon sees an area larger than the current, more inserted field ofview. A kinematically accurate graphical simulation of the instrumentsis shown in the old field of view 2942 so that the surgeon can seepossible collisions in this region as the instruments move.

FIG. 21 is a diagrammatic perspective view that shows aspects of anillustrative minimally invasive surgical instrument assembly thatincludes a multi jointed instrument dedicated to retraction. As shown inFIG. 21, guide tube 3102 includes a channel 3104, through which animaging system is inserted, and three channels 3106 a,3106 b,3106 c,through which surgical instruments may be inserted. Retractioninstrument 3108 is shown extending through channel 3106 c.

As depicted, retraction instrument 3108 includes a proximal instrumentbody 3110 and four serial links 3112 a-d. Four joints 3114 a-d coupleproximal instrument body 3110 and links 3112 a-d together. In oneaspect, each joint 3114 a-d is an independently controllable single DOFpitch joint. In other aspects the joints may have additional DOFs. Anactively controlled (either hand or telemanipulated) gripper 3116 ismounted at the distal end of the most distal link 3112 d via a passiveroll joint 3118. In some aspects other end effectors, or none, may besubstituted for the gripper. In one aspect the combined length of links3112 a-d and gripper 3116 is sufficient to retract tissue beyond theworking envelope of instruments that extend through channels 3106 a and3106 b. For example, the combined lengths of the links and the grippermay be approximately equal to the full insertion range (e.g.,approximately 5 inches) of the instruments. Four links and joints areshown, and other numbers of links and joints may be used. Retraction isdone using various combinations of pitching joints 3114 a-d and rollinginstrument 3108 within channel 3106 c.

For performing a retraction, instrument 3108 is inserted so that eachjoint 3114 a-d is exposed one after the other. Insertion depth may bevaried so that retraction can begin at various distances from the distalend of the guide tube with various numbers of joints as the joints exitfrom the guide tube's distal end. That is, for example, retraction maybegin as soon as joint 3114 d is inserted past the distal end of theguide tube. For retraction, gripper 3116 may grip tissue. Passive rolljoint 3118 prevents the gripped tissue from being torqued as instrument3108 is rolled within channel 3106 c. In one aspect, the control systemcouples the motions of instrument 3108 and guide tube 3102. This coupledcontrol of motion allows tissue to be held in place by gripper 3116 asthe guide tube is moved to the left or right “underneath” the retractedtissue. For example, as the distal end of guide tube 3102 is moved tothe left, instrument 3108 is rolled (and joint 3114 a-d pitch may bechanged) to move gripper 3116 to the right.

FIG. 21 further illustrates an aspect of instrument position and controlwithin guide tubes. The working surgical instruments need not beinserted though guide tube channels that correspond to or are alignedwith their working position. For example, as shown in FIG. 31 the leftside working instrument does not have to be inserted through theleft-most channel 3106 c. Instead, the left side working instrument maybe inserted via the “bottom” channel 3106 b. The right side workinginstrument may then be inserted via the right-most channel 3106 a. Then,the left and right side working instruments may be controlled to work ata surgical site in alignment with the field of view of an imaging systeminserted via channel 3104 that has not been rolled or yawed. Statedanother way, the left-right axis between the working instruments'insertion channels does not have to be aligned with the left-right axisbetween the working instruments' end effectors at the surgical site orwith the left-right axis interpupillary axis of the stereoscopic imagingsystem. Further, by the control system recognizing which instrument iscoupled to each particular actuator, left-right instrument position maybe varied. For example, retraction instrument 3108 may be inserted viachannel 3106 a, the right side working instrument may be inserted viachannel 3106 b, and the left side working instrument may be inserted viachannel 3106 c. In some aspects, with appropriately shaped channelsand/or imaging systems, the imaging system may be inserted via one ofseveral channels. For example, “top” channel 3104 and “bottom” channel3106 b may be oblong shaped with a center bore that holds a cylindricalinstrument body. Consequently, an imaging system may be inserted via the“top” or “bottom” channel, and a working instrument may be inserted viathe other “top” or “bottom” channel.

FIG. 22 is a block diagram of components used for controlling andselectively associating medical devices on the patient side supportsystem 2104 to operator manipulated input devices 203, 204 of thesurgeon's console 2102. Various surgical tools such as graspers,cutters, and needles may be used to perform a medical procedure at awork site within the Patient. In this example, three surgical tools(TOOL1,TOOL2,TOOL3) 2231,2241,2251 are used to robotically perform theprocedure and the imaging system (IS) 2261 is used to view theprocedure. The tools 2231,2241,2251 and imaging system 2261 may bedisposed in a guide tube (GT) 2271 so as to be extendable beyond adistal end of the guide tube 2271. The guide tube 2271 may be insertedinto the Patient through an entry aperture such as a minimally invasiveincision or a natural orifice using the setup portion of a robotic armassembly and maneuvered by a guide tube manipulator 2272 towards thework site where the medical procedure is to be performed.

Each of the devices 2231,2241,2251,2261,2271 is manipulated by its ownmanipulator. In particular, the imaging system 2261 is manipulated by animaging system manipulator (PSM4) 2262, the first surgical tool 2231 ismanipulated by a first tool manipulator (PSM1) 2232, the second surgicaltool 2241 is manipulated by a second tool manipulator (PSM2) 2242, thethird surgical tool 2251 is manipulated by a third tool manipulator(PSM3) 2252, and the guide tube 2271 is manipulated by a guide tubemanipulator 2272.

Each of the instrument manipulators 2232,2242,2252,2262 is a mechanicalassembly that carries actuators and provides a mechanical, sterileinterface to transmit motion to its respective articulated instrument.Each instrument 2231,2241,2251,2261 is a mechanical assembly thatreceives the motion from its manipulator and, by means of a cabletransmission, propagates the motion to its distal articulations (e.g.,joints). Such joints may be prismatic (e.g., linear motion) orrotational (e.g., they pivot about a mechanical axis). Furthermore, theinstrument may have internal mechanical constraints (e.g., cables,gearing, cams, belts, etc.) that force multiple joints to move togetherin a pre-determined fashion. Each set of mechanically constrained jointsimplements a specific axis of motion, and constraints may be devised topair rotational joints (e.g., joggle joints). Note also that in this waythe instrument may have more joints than the available actuators.

In direct control mode, each of the input devices 203,204 may beselectively associated with one of the devices 2261,2231,2241,2251, 2271through a multiplexer (MUX) 2290 so that the associated device may becontrolled by the input device through its controller and manipulator.For example, the Surgeon may specify the association through a graphicaluser interface (GUI) 2291 on the surgeon's console 2102 for the left andright input devices 203,204 to be respectively associated with the firstand second surgical tools 2231, 2241, which are teleroboticallycontrolled through their respective controllers 2233, 2243 andmanipulators 2232,2242 so that the Surgeon may perform a medicalprocedure on the Patient while the surgical tool 2251, imaging system2261 and guide tube 2271 are each soft locked in place through theirrespective controllers (such as shown in FIGS. 24, 25). If the Surgeondesires to control movement of the surgical tool 2251 using one of theinput devices 203,204, then the Surgeon may do so by simplydisassociating the input device from its currently associated device andassociating it instead to the tool 2251. Likewise, if the Surgeondesires to control movement of either the imaging system 2261 or guidetube 2271 using one or both of the input devices 203,204, then theSurgeon may do so by simply disassociating the input device from itscurrently associated device and associating it instead to the imagingsystem 2261 or guide tube 2271.

As alternatives to using the GUI 2291 for providing selection input SELfor the MUX 2290, the selective association of the input devices 203,204to devices 2251,2241,2231,2261,2271 may be performed by the Surgeonusing voice commands understood by a voice recognition system, or by theSurgeon depressing a button on one of the input devices 203,204, or bythe Surgeon depressing a foot pedal on the surgeon's console 2102, or bythe Surgeon using any other well known mode switching technique.Although such mode switching is described herein as being performed bythe Surgeon, it may alternatively be performed by an Assistant under thedirection of the Surgeon.

Each of the controllers 2233,2243,2253,2263,2273 comprises amaster/slave control system. FIG. 23 illustrates, as an example, a blockdiagram of a master/slave control system 300 for controlling movement ofthe tool slave manipulator 2232 when it is associated with the inputdevice 203 and consequently, the position and orientation of itsattached tool 2231, as commanded by movement of the master manipulator203 by the Surgeon. A similar master/slave control system may beprovided for each of the other slave manipulators (e.g.,2241,2251,2261,2271) in the system 2100.

Both the master and slave manipulators include a number of linkagesconnected by joints so as to facilitate multiple degrees-of-freedommovement. As the Surgeon moves the master manipulator 203 from oneposition to another during the course of performing a surgicalprocedure, sensors associated with the master manipulator joints provideinformation indicating such command movement in master joint space, andsensors associated with the slave manipulator joints provide informationindicating slave manipulator and consequently, tool 2231 movement inslave joint space for feedback purposes.

A master input processing unit 301 receives the information of themaster joint positions, which are sampled at the control systemprocessing rate (e.g., 1300 Hz in the present example), from the masterjoint sensors in the master manipulator 203, and computes jointvelocities from the sensed joint positions. A master forward kinematicsprocessing unit 302 receives the master joint positions and velocitiesfrom the master input processing unit 301, transforms them from masterjoint space to corresponding positions and velocities of the masterframe (i.e., the frame associated with the master manipulator 203) inCartesian space relative to the eye reference frame (i.e., the referenceframe associated with the position of the surgeon's eyes), using, forexample, a Jacobian matrix and eye related information separatelydetermined and provided in block 303.

A scale and offset processing unit 304 receives the Cartesian positionand velocity commands from the master forward kinematics processing unit302, scales the commanded movement according to a scale factor selectedto perform the surgical procedure, and takes into account offsets togenerate desired slave tool frame (i.e., the frame associated with thetool 2231) positions and velocities. For economy of words, Cartesianposition is to be interpreted to include Cartesian orientation in thisspecification where appropriate, Cartesian velocity is to be interpretedto include translational and angular velocities where appropriate. Thescale adjustment is useful where small movements of the slavemanipulator 2232 are desired relative to larger movement of the mastermanipulator 203 in order to allow more precise movement of the slavetool 2231 at the surgical site. The offsets, on the other hand,determine, for example, the corresponding position and/or orientation ofan end effector frame (e.g., the frame associated with an end effectorof the tool 2231) in the camera reference frame (i.e., the frameassociated with the image capturing end of the imaging system) relativeto a position and orientation of the master frame in the eye referenceframe.

A simulated slave processing unit 308 (also referred to as a “simulateddomain”) receives desired slave tool frame position and velocitycommands from the scale and offset processing unit 304, and limits thedesired slave tool frame position, orientation and velocities, toassigned Cartesian limits for instance to enforce correct and intuitiveoperation of the tool 2231 by keeping it within its dexterous workspaceand to prevent motions that would result in excessive forces beingexerted by the end effector of the tool 2231. The simulated slaveprocessing unit 308 generates simulated slave joint positions andvelocities corresponding to the limited slave tool frame positions andvelocities, while making sure that the generated slave joint positionsand velocities do not exceed the actual slave joint's range of motionand maximum velocities (i.e., joint limits) even in the vicinity ofkinematic singularities for the slave kinematics.

An inverse scale and offset processing unit 306 receives the simulatedjoint position and velocity commands from the simulated slave processingunit 308, and performs an inverse function to that of the scale andoffset processing unit 304 on them. A Cartesian controller 307 receivesas first inputs, the inputs to the scale and offset processing unit 304and as second inputs, the outputs of the inverse scale and offsetprocessing unit 306. The Cartesian controller 307 then generates anerror signal as a difference of the first and second inputs, and aCartesian force “F_(CART)” from the error signal such as with thefollowing formula:F _(CART) =K(Δx)+B(Δ{dot over (x)})  (1)where “K” is a spring constant, “B” is a damping constant, “Δ{dot over(x)}” is the difference between the Cartesian velocity inputs to theCartesian controller 307 and “Δx” is the difference between theCartesian position inputs to the Cartesian controller 307. For anorientation error, a corresponding torque in Cartesian space isdetermined.

A master transpose kinematics processing unit 315 receives the Cartesianforce F_(CART) through a summation node 314, and generates acorresponding torque in joint space using, for example, the Jacobiantranspose matrix and kinematic relationships associated with the mastermanipulator 203. A master output processing unit 316 receives the mastertorque signals from the master transpose kinematics processing unit 315,generates electrical currents corresponding to the master torquesignals, and supplies the electrical currents to corresponding masterjoint motors of the master manipulator 203. As a result, a surgeonoperating the master manipulator 203 feels the Cartesian force,F_(CART), whenever the surgeon is commanding a position or velocitywhich exceeds system Cartesian or slave joint limits, or would result ina kinematic singularity condition for the slave manipulator 2232.

As the master input processing unit 301 is receiving master jointpositions from sensors in the master manipulator 203, a slave inputprocessing unit 309 is also receiving slave joint positions fromposition sensors in the slave manipulator 2232 at the control systemprocessing rate. A joint control unit 320 receives the slave jointpositions from the slave input processing unit 309 and the simulatedjoint position commands provided from the simulated slave processingunit 308, and generates slave torque command signals for the slave jointmotors and master torque feedback command signals for the master jointmotors.

The slave torque command signals are generated by the joint control unit320 so as to drive joints of the slave manipulator until feedback errorscalculated in the joint control unit 320 zero out. A slave outputprocessing unit 310 receives the slave torque command signals from thejoint control unit 320, converts them into appropriate electricalcurrents, and supplies the electrical currents to the joint motors ofthe slave manipulator so as to drive the motors accordingly.

The master torque feedback command signals are generated by the jointcontrol unit 320 as a function of the slave joint position and velocitytracking errors so as to reflect forces being exerted against the tool2231 or its slave manipulator 2232 back to the master manipulator 203 sothat they may be felt by the Surgeon. A kinematic mapping unit 311receives the master torque feedback command signals from the jointcontrol unit 320, and generates the corresponding Cartesian force beingexerted against the tip of the tool 2231 relative to the camera frame ofthe imaging system using the slave kinematic configuration and thepreviously calculated slave reference frame position informationprovided in block 312.

A gain 313 adjusts the magnitude of the Cartesian force so as to ensuresystem stability while providing adequate force sensation to theSurgeon. The gain adjusted Cartesian force is then passed through thesummation node 314, and processed along with the Cartesian forceprovided by the Cartesian controller 307 through the master transposekinematics processing unit 315 and master output processing 316 aspreviously described in reference to their processing of the Cartesianforce provided by the Cartesian controller 307.

Additional details related to conventional aspects of the master/slavecontrol system 300, such as the various reference frames referred toherein and the calculation of the surgeon eye related informationprovided in block 303 and the slave reference frame information providedin block 312, which are based upon well-known mathematics, aredescribed, for example, in previously incorporated by reference and U.S.Pat. No. 6,424,885, “Camera Referenced Control in a Minimally InvasiveSurgical Apparatus” where the notion of reference frame is termed “slavefulcrum”.

The joint control unit 320 includes a joint controller for each activejoint and gear of the slave manipulator 2232 that is being controlled bythe master/slave control system 300. In particular, where the slavemanipulator 2232 includes various joints to move the tool 2231 throughits operable workspace, each of these joints will have its owncontroller. To simplify the description herein and in the claims, theterm “joint” is to be understood as a connection (translational orrevolute) between two links, and may include gears (or prismatic joints)as well as any other controllable component coupled to linear drivemechanisms that may be used in controlling robotic arm assemblies.

Direct control modes are control modes in which the user has directcontrol over a specific slave manipulator. All other slave manipulators(i.e., the ones that are not connected to a master device) aresoft-locked (i.e., all their joints are held in place by theirrespective controllers). As an example, in a single-port system such asdescribed herein, three direct control modes are defined as a direct“tool following” mode in which the two hand-operable input devices areassociated with two tool slave manipulators and their respective tools,a direct “imaging system” mode in which one or both of the hand-operableinput devices are associated with the imaging system, and a direct“guide tube” mode in which one or both hand-operable input devices areassociated with the guide tube. For examples, FIGS. 24-25 illustrate adirect “tool following” mode in which the left and right master inputdevices 204,203 are respectively associated with the first and secondtools while a third tool, the imaging system and the guide tube are heldin place by their respective controllers; FIGS. 26-27 illustrate adirect “imaging system” mode in which the left master input device 204is associated with the imaging system while the first tool, second tool,third tool and guide tube are held in place by their respectivecontrollers; and FIGS. 28-29 illustrate a direct “guide tube” mode inwhich the left and right master input devices 204,203 are associatedwith the guide tube while the first tool, second tool, third tool andimaging system are held in place by their respective controllers.

As indicated in FIGS. 24,26,28, data pick-off/receiving points(respectively at the inputs to the inverse scale & offset blocks 306 andoutputs of the scale & offset blocks 304 of the master/slave controlsystems implemented in the associated device controllers) are availableto provide commanded state information to non-associated controllers forcoupled control modes and receive state information back from thenon-associated controllers, as described herein. To simplify thedrawings, both the inverse scale & offset block 306 and scale & offsetblock 304 are included in a single block designated as “Scale & Offset”in the figures. Although data pick-off/receiving points respectively atthe inputs to the inverse scale & offset blocks 306 and outputs of thescale & offset blocks 304 are used in these examples, it is to beappreciated that other data pick-off and receiving points may be used inpracticing the various aspects of the present invention.

Also to simplify the figures, the master/slave control system 300 hasbeen split into master and slave side portions (on opposite sides of the“Scale & Offset” blocks) with the PSM1* Controller 248, PSM2* Controller247, PSM4* Controller 268, and GT* Controller 288 comprising the slaveside components (e.g., control system 300 blocks308,320,309,310,311,312,313 of FIG. 23) and the MTM Controllers241,242,262,281,282 comprising the master side components (e.g., controlsystem 300 blocks 301,302,303,307,314,315,316 of FIG. 23). Hold positionblocks 251,252,253,271,272 in FIGS. 25, 27, 29 indicate state commands(each indicating a constant position and orientation for its respectivedevice) that are stored in one or more memory devices and respectivelyprovided to the slave side PSM3* Controller 258, GT* Controller 288,PSM4* Controller 268, PSM1* Controller 248, and PSM2* Controller 247,while following data generated in these controllers are ignored (orotherwise discarded) as indicated by downward point arrows from thesecontrollers, so that their respective manipulators and devices are heldat the commanded states.

In a coupled control mode, the Surgeon is directly controlling movementof an associated slave manipulator (e.g., one of the manipulators2232,2242,2252,2262,2272) while indirectly controlling movement of oneor more non-associated slave manipulators, in response to commandedmotion of the directly controlled slave manipulator, to achieve asecondary objective. Examples of secondary objective include optimizingdevice workspaces (i.e., maximizing their ranges of motion), optimizingthe imaging system's view of other devices and/or the work site,minimizing the chance of collisions between devices and/or the patient'sanatomy, and driving non-associated devices to desired poses. Byautomatically performing secondary tasks through coupled control modes,the system's usability is enhanced by reducing the Surgeon's need toswitch to another direct mode to manually achieve the desired secondaryobjective. Thus, coupled control modes allow the Surgeon to better focuson performing the medical procedure and to pay less attention tomanaging the system. As described below, the user interface has threecoupled control modes: a mode for the instrument(s), a mode for theimaging system, and a mode for guide tube (i.e. as many modes as thenumber of manipulators designed to perform different functions withinthe surgical system).

It is useful to provide haptic cues to the Surgeon to indicate whenmotion of a coupled manipulator occurs, since the Surgeon otherwise maynot be aware of the movement of any device that is being indirectlycontrolled through a coupled control mode. This is not a problem fordirectly controlled devices, because the master/slave control system forsuch directly controlled devices generally provides a haptic feedbackpath. Therefore, a haptic cue such as a detent may be provided thatsignals to the Surgeon when a coupled mode becomes engaged.

The GUI 2291 used by the Surgeon to specify the association of inputsdevices 203,204 and devices 2231,2241,2251,2261,2271 may also be used bythe Surgeon to specify various parameters of the coupled control modes.For example, the Surgeon may use the GUI 2291 to select which devicemanipulators participate in various coupled control modes and to defineand/or prioritize the secondary objectives associated with the coupledcontrol modes.

FIG. 30 is a diagrammatic view that illustrates coupled control aspectsof a centralized motion control and coordination system architecture forminimally invasive telesurgical systems that incorporate surgicalinstrument assemblies and components described herein. A motioncoordinator system 2202 receives master inputs 2204, sensor inputs 2206,and optimization inputs 2208.

Master inputs 2204 may include the surgeon's arm, wrist, hand, andfinger movements on the master control mechanisms. Inputs may also befrom other movements (e.g., finger, foot, knee, etc. pressing or movingbuttons, levers, switches, etc.) and commands (e.g., voice) that controlthe position and orientation of a particular component or that control atask-specific operation (e.g., energizing an electrocautery end effectoror laser, imaging system operation, and the like).

Sensor inputs 2206 may include position information from, e.g., measuredservomotor position or sensed bend information. U.S. patent applicationSer. No. 11/491,384 (Larkin, et al.) entitled “Robotic surgery systemincluding position sensors using fiber Bragg gratings”, incorporated byreference, describes the use of fiber Bragg gratings for positionsensing. Such bend sensors may be incorporated into the variousinstruments and imaging systems described herein to be used whendetermining position and orientation information for a component (e.g.,an end effector tip). Position and orientation information may also begenerated by one or more sensors (e.g., fluoroscopy, MM, ultrasound, andthe like) positioned outside of the patient, and which in real timesense changes in position and orientation of components inside thepatient.

Optimization inputs 2208 relate to the secondary objectives. They may behigh-level commands, or the inputs may include more detailed commands orsensory information. An example of a high level command would be acommand to an intelligent controller to optimize a workspace. An exampleof a more detailed command would be for an imaging system to start orstop optimizing its camera. An example of a sensor input would be asignal that a workspace limit had been reached.

Motion coordinator 2202 outputs command signals to various actuatorcontrollers and actuators (e.g., servomotors) associated withmanipulators for the various telesurgical system arms. FIG. 30 depictsan example of output signals being sent to two instrument controllers2210, to an imaging system controller 2212, and to a guide tubecontroller 2214. Other numbers and combinations of controllers may beused. The motion coordinator 2202 determines how to take advantage ofthe overall system kinematics (i.e., the total degrees of freedom of thesystem) to achieve the secondary objectives indicated by theoptimization inputs 2208.

As an example, such a motion coordination system may be used to controlsurgical instrument assembly 1700 (FIG. 6). Instrument controllers 2210are associated with instruments 1702 a,1702 b, imaging system controller2212 is associated with imaging system 1704, and guide tube controller2214 is associated with guide tube 1708. Accordingly, in some aspectsthe surgeon who operates the telesurgical system will simultaneously andautomatically access at least the three control modes identified above:an instrument control mode for moving the instruments, an imaging systemcontrol mode for moving the imaging system, and a guide tube controlmode for moving the guide tube. A similar centralized architecture maybe adapted to work with the various other mechanism aspects describedherein.

FIG. 31 is a diagrammatic view that illustrates aspects of a distributedmotion control and coordination system architecture for minimallyinvasive telesurgical systems that incorporate surgical instrumentassemblies and components described herein. In the illustrative aspectsshown in FIG. 31, control and transform processor 2220 exchangesinformation with two master arm optimizer/controllers 2222 a,2222 b,with three surgical instrument optimizer/controllers 2224 a,2224 b,2224c, with an imaging system optimizer/controller 2226, and with a guidetube optimizer/controller 2228. Each optimizer/controller is associatedwith a master or slave arm (which includes, e.g., the camera (imagingsystem) arm, the guide tube arm, and the instrument arms) in thetelesurgical system. Each of the optimizer/controllers receivesarm-specific optimization goals 2230 a-2230 g.

The double-headed arrows between control and transform processor 2220and the various optimizer/controllers represents the exchange ofFollowing Data associated with the optimizer/controller's arm. FollowingData includes the full Cartesian configuration of the entire arm,including base frame and distal tip frame. Control and transformprocessor 2220 routes the Following Data received from eachoptimizer/controller to all the optimizer/controllers so that eachoptimizer/controller has data about the current Cartesian configurationof all arms in the system. In addition, the optimizer/controller foreach arm receives optimization goals that are unique for the arm. Eacharm's optimizer/controller then uses the other arm positions as inputsand constraints as it pursues its optimization goals. In one aspect,each optimization controller uses an embedded local optimizer to pursueits optimization goals. The optimization module for each arm'soptimizer/controller can be independently turned on or off. For example,the optimization module for only the imaging system and the guide tubemay be turned on.

The distributed control architecture provides more flexibility than thecentralized architecture, although with the potential for decreasedperformance. It easier to add in a new arm and to change the overallsystem configuration if such a distributed control architecture is usedrather than if a centralized architecture is used. In this distributedarchitecture, however, the optimization is local versus the globaloptimization that can be performed with the centralized architecture, inwhich a single module is aware of the full system's state.

FIGS. 32-34 illustrate aspects of particular coupled control modes whereassociated devices are directly controlled to accomplish primaryobjectives and non-associated devices are indirectly controlled toaccomplish secondary objectives. In particular, FIG. 32 illustrates acoupled “tool following” mode example in which the left and right masterinput devices 204,203 are respectively associated with the first andsecond tools while information of their commanded movement is madeavailable by coupling blocks 3202,3201 connected to data pick-off pointsof their respective master/slave control systems to coupled controllers3204,3203 of the imaging system and the guide tube so that they mayperform desired “secondary” objectives; FIG. 33 illustrates a coupled“imaging system” mode example in which the left master input device 204is associated with the imaging system while information of its commandedmovement is made available by a coupling block 3302 connected to datapick-off points of its master/slave control system to coupledcontrollers 3304,3305,3303 of the first tool, second tool, and guidetube so that they may perform desired “secondary” objectives; and FIG.34 illustrates a coupled “guide tube” mode example in which the left andright master input devices 204,203 are associated with the guide tubewhile information of its commanded movement is made available by acoupling block 3402 connected to data pick-off points of itsmaster/slave control system to coupled controllers 3404,3405,3403 of thefirst tool, second tool, and imaging system so that they may performdesired “secondary” objectives. Note that in these coupled modeexamples, the third tool is assumed not to be deployed to simplify thefigures.

The coupling blocks and device coupled controllers illustrated in FIGS.32-34 may be implemented in a distributed fashion as shown so that theyare either integrated in or implemented outside their respectivecontrollers or they may be implemented in a centralized fashion so theyare integrated into a single unit outside of their respectivecontrollers. To transform the direct “tool following” mode to acorresponding coupled “tool following” mode, coupling blocks 3201,3202(as shown in FIG. 32) are coupled to the data pick-off/receiving points(as shown in FIG. 24). Device coupled controllers 3203,3204, whichcommand and control their respective device controllers to performsecondary objectives, are coupled at data pick-off/receiving points oftheir respective device controllers and to the coupling blocks 3202,3201so that they may receive and send information back and forth asindicated by the arrows in FIG. 32. Likewise, to transform the direct“imaging system” mode to a corresponding coupled “imaging system” mode,coupling block 3302 (as shown in FIG. 33) is coupled to the datapick-off/receiving points (as shown in FIG. 26). Device coupledcontrollers 3303,3304,3305, which command and control their respectivedevice controllers to perform secondary objectives, are coupled at datapick-off/receiving points of their respective device controllers and tothe coupling block 3302 so that they may receive and send informationback and forth as indicated by the arrows in FIG. 33. Finally, totransform the direct “guide tube” mode to a corresponding coupled “guidetube” mode, coupling block 3402 (as shown in FIG. 34) is coupled to thedata pick-off/receiving points (as shown in FIG. 28). Device coupledcontrollers 3403,3404,3405, which command and control their respectivedevice controllers to perform secondary objectives, are coupled at datapick-off/receiving points of their respective device controllers and tothe coupling block 3402 so that they may receive and send informationback and forth as indicated by the arrows in FIG. 34.

FIGS. 35-40 are flow diagrams illustrating examples of coupled controlmodes. As previously explained, the user interface has three coupledcontrol modes: a mode for the instrument(s), a mode for the imagingsystem, and a mode for the guide tube. FIGS. 35-37 and 39 are examplesof instrument coupled control, FIG. 38 is an example of a guide tubecoupled control, and FIG. 40 is an example of an imaging system coupledcontrol. The methods described in reference to FIGS. 35-40, as well asvarious controllers and other processing units described herein arepreferably implemented in processor 220 as described in reference toFIG. 4.

FIGS. 35-36 illustrate a first part of an example of instrument coupledcontrol in which the workspaces of articulated devices that are coupledto a guide tube and extendable beyond a distal end of the guide tube areoptimized. FIG. 6 is one example of such an instrument assembly.Although the present example describes use of a guide tube foroptimizing workspaces of articulated devices used to perform a medicalprocedure, it is to be appreciated that aspects of the invention arealso applicable to any base coupled to the articulated devices so thatthe articulated devices move when the base moves. As an example, thepatient side support system 2104, which is described in reference toFIGS. 1-3, may also function as such a base if rotational setup joints2114 a, 2114 b are actively drivable.

Referring first to FIG. 35, in 3501, the guide tube coupled controller3203 (e.g., the motion coordinator 2202 or the guide tubeoptimizer/controller 2228, depending upon whether a centralized ordistributed coupled mode architecture is employed), that is operating atthe time in an instrument coupled control mode, receives commandeddevice tip positions from coupling blocks and coupled controllers of alldevices that are coupled to the guide tube (i.e., devices that move whenthe guide tube moves). For example, a device may be coupled to the guidetube if it is disposed within the guide tube or if it is otherwisephysically attached to the guide tube. As used herein (except in placeswhere the context of the description clearly indicates otherwise), thephrase “device tip position” means information indicative of theCartesian coordinates in a fixed reference frame for the device's mostdistal joint and an orientation determined by an angular position of themost distal joint.

In 3502, the guide tube coupled controller 3203 uses the receivedcommanded device tip positions to determine a guide tube tip positionthat optimizes workspaces of the devices coupled to the guide tube whiletheir respective controllers maintain their device tip positions. Sincethe optimization function requires knowledge of the range of motionlimits and kinematics of the devices, as well as the current tippositions of the guide tube and the devices, such range of motion andkinematics information is preferably provided to the guide tube coupledcontroller 3203 either at system startup or other convenient time in aconventional manner while current tip positions of the devices areprovided during operation by the device coupling blocks and coupledcontrollers as previously described. To determine the desired guide tubetip position, each of the device controllers may provide a desiredCartesian pose for its device so that the guide tube coupled controllersolves the kinematics in such a way as to have the guide tube tippositioned so as to allow the device's joints to be configurable asclose as possible to its desired pose while not moving its tip from thedesired tip position.

Preferably such optimization is performed by minimizing a cost functionusing ranges of motion of the devices and selected weightings. Forexample, weight values may be selected so that maximizing the ranges ofmotions of the instruments 2231,2241 being directly controlled is moreheavily weighted (i.e., having higher priority) than maximizing therange of motion of the imaging system 2261 and any other device whosetip is being held in place (i.e., held or “soft-locked” in position byits controller). In 3503, the determined guide tube tip position is thenprovided to the guide tube controller 2272 to drive the guide tube 2271to the determined tip position and to the device controllers2233,2243,2263 so that they may drive their respective devices2231,2241,2261 to articulated joint configurations that optimize theirrespective workspaces as described in reference to FIG. 36 as follows.

Referring now to FIG. 36 to describe complementary actions performed bythe device controllers of devices coupled to the guide tube, in 3601,the commanded position of the guide tube 2271 is received from the guidetube coupled controller. In 3602, the device controllers generateupdated joint position commands for their respective slave manipulatorsto accommodate the new guide tube position while satisfying commandeddevice tip positions. For the instruments 2231,2241, which areassociated with the input devices 204,203 under instrument coupledcontrol mode, the commanded device tip positions correspond to tippositions commanded by the input devices 204,203. For the imaging system2261 or another instrument 2251, which are not associated at the timewith the input devices 204,203, the commanded device tip positions aretheir current tip positions so that the tips of these non-associateddevices are effectively held in place. In 3603, the device controllersprovide the updated joint position commands to their respective slavemanipulators so that the device workspaces are optimized.

FIG. 37 illustrates an optional second part of the example in whichmovement of the imaging system 2261 is coupled to movement of theinstruments 2231,2241 so that the instruments are well placed in a fieldof view of the imaging system. Whereas the first part of the exampledescribed in reference to FIGS. 35-36 addresses the secondary objectiveof optimizing the workspaces of devices coupled to the guide tube andextendable beyond the distal end of the guide tube, the second part ofthe example addresses the secondary objective of optimizing the view ofthe device tips in images captured by the imaging system.

Referring now to FIG. 37, in 3701, the imaging system coupled controller3204 (e.g., the motion coordinator 2202 or the imaging systemoptimizer/controller 2226, depending upon whether a centralized ordistributed coupled mode architecture is employed), that is operating atthe time in an instrument coupled control mode, receives commandeddevice tip positions from all device controllers. In 3702, the imagingsystem coupled controller determines a centroid of the commandedinstrument tip positions and in 3703, it determines a centroid velocityusing differences in centroid positions determined in the present andprevious digital process periods. In 3704 and 3705, tremor filtering isperformed to determine a desired imaging system tip position andvelocity by respectively applying a dead zone behavior to the centroidposition and a low pass filter to the centroid velocity.

In 3706, the imaging system coupled controller 3204 then determinesdesired joint positions for the imaging system 2261 using inversekinematics of the articulated imaging system 2261 and the current tipposition of the guide tube 2271. In 3707, the imaging system coupledcontroller determines an imaging system tip position corresponding tothe modified slave joint positions using forward kinematics of theimaging system 2261 and provides the determined imaging system tipposition to the guide tube coupled controller 3203. Note that theimaging system tip position determined in 3707 should be the same as thedesired imaging system tip position in 3704 unless joint limits orsingularities were encountered in 3707, in which case, they would bedifferent in order to avoid the limits or singularities. The guide tubecoupled controller then processes the imaging system tip position alongwith the instrument tip positions according to the first part of theexample as described in reference to FIG. 35 to generate a guide tubetip position that optimizes workspaces of the instruments and imagingsystem. In 3708, the imaging system coupled controller 3204 receives thecommanded guide tube tip position from the guide tube coupled controller3203 and uses it in 3709 to determine commanded slave joint positions byapplying the imaging system tip position determined in 3707 and themodified guide tube tip position to the same equations and limits usedin performing 3706. In 3710, the commanded slave joint positionsdetermined by the imaging system controller 2263 are then provided asactuator commands to the imaging system manipulator 2262 to manipulateor move the imaging system 2261 accordingly.

Upon completion of a medical procedure, all medical devices used duringthe procedure should be retracted back out of the patient. Rather thandoing this one at a time using direct control modes, it is advantageousto retract all devices at the same time using coupled control modes. Inparticular, by retracting one device under direct control, it isdesirable that all other devices follow in retraction under coupledcontrol while addressing secondary objectives such as avoidingcollisions with each other and/or the patient anatomy during theretraction. In addition, before retracting each device into its guidetube, it is necessary to first place the device in a retractionconfiguration so that it may be retracted into the guide tube. Forexample, the retraction instrument 3108 depicted in FIG. 21 can only befully retracted into channel 3106 c of guide tube 3102 after each of itslinks 3112 a-3112 d is aligned with the channel 3106 c. Thus, it isdesirable to automatically drive each of the devices into its retractionconfiguration before the device enters its guide tube. This applies tothe device being retracted under direct control as well as the devicesbeing retracted indirectly through coupled control modes.

Conversely, before performing a medical procedure, all medical devicesto be used during the procedure should be inserted into the patient.Rather than doing this one at a time using direct control modes, it isadvantageous to insert all devices at the same time using coupledcontrol modes. In particular, by inserting one device under directcontrol, it is desirable that all other devices follow in insertionunder coupled control while addressing secondary objectives such asavoiding collisions with each other and/or the patient anatomy duringthe insertion. In addition, after the instruments are inserted into thepatient and they reach the work site, it is useful to place theinstruments into configurations that optimize their workspaces. It isalso useful for the working ends of the instruments to be well placed ina field of view of an imaging system. Thus, it is desirable toautomatically drive each of the instruments into its optimalconfiguration after the imaging system reaches a desired viewing pointat the work site.

FIG. 38 illustrates an example of using coupled control for retractingmedical devices into a guide tube. Although any one of the devices maybe directly controlled while the others are indirectly controlled forretraction into the guide tube, the example employs a virtual degree offreedom (DOF) of the guide tube manipulator for controlling theretraction. Since all devices are coupled to the guide tube, all devicesmove as the guide tube moves. The guide tube manipulator in thisexample, however, doesn't have an actuator for insertion/retraction, soit effects a virtual insertion/retraction DOF by causing the devices tobe moved in the desired insertion/retraction direction by passing theguide tube insertion/retraction command to each of the devicecontrollers while the guide tube remains in place.

In 3801, the guide tube coupling block 3402 periodically receivesconventional time-sampled output from its associated Surgeon manipulatedinput device(s) that indicates in this case that the guide tube is to beretracted backward (e.g., away from a work site) along its longitudinalaxis. In 3802, the coupling block 3402 relays the received retractioncommands to the other device coupled controllers so that they in turn,command their respective device manipulators to retract their respectivedevices in the desired retraction direction from their positions at thetime.

In 3803, each of the device controllers (i.e., other than the guide tubecontroller) determines when the proximal end of the most proximalrotated link of its respective device is within a threshold distance“TH” from the distal end of the guide tube. The threshold distance “TH”may be determined, for example, by taking into account the currentrotation angle of the most proximal rotated link, the rate at which theretraction is being commanded by the Surgeon on the input device, andthe clearance between the “straightened out” device and the channelthrough which the device extends through in the guide tube. Inparticular, the threshold distance “TH” is selected so that each of thedevices may be retracted back into the guide tube without striking theends or sides of its respective channel through which it is disposed.

The distance between the proximal end of the most proximal rotated linkof device and the distal end of the guide tube may be determined in aconventional manner by determining a first vector that extends from aremote center “RC” (i.e., a pivot point of the guide tube) to the distalend of the guide tube, determining the most proximal rotated link of thedevice, determining a second vector that extends from the remote center“RC” to the most proximal joint rotating the most proximal rotated linkof the device, and determining the distance between the proximal end ofthe most proximal rotated link of a device and the distal end of theguide tube from the difference between the first and second vectors.

In 3804, each of the device controllers (i.e., other than the guide tubecontroller) drives its device to a retraction configuration (i.e., ajoint and link configuration that allows the device to be fullyretracted into the guide tube) upon determining that the proximal end ofthe most proximal rotated link of its respective device is within thethreshold distance “TH” from the distal end of the guide tube. The ratethat the device is driven to its retraction configuration is determinedat least in part by the rate at which the output of the input device ischanging in the insertion/retraction commanded direction so thatcollisions between the device and the guide tube are avoided. Inaddition, possible collisions with other devices and/or the patient arealso to be avoided and taken into account as each of the devicecontrollers drives its device to its retraction configuration. In 3805,once each device is determined by its respective device controller to bein its retraction configuration, the device controller allows itsrespective device to be retracted into its channel in the guide tube inresponse to retraction commands issued from the input device(s)associated at the time with the guide tube.

Since the image capturing end of the imaging system is generallypositioned closer to the distal end of the guide tube than theinstruments so that the working ends of the instruments and the worksite are well positioned within the field of view of the imaging system,the most proximal rotated link of the imaging system will generally bethe first rotated link of the group of devices extending beyond thedistal end of the guide tube to reach the threshold distance “TH” fromthe distal end when the group of devices is being retracted. As the mostproximal rotated link of each of the other devices reaches the thresholddistance “TH” from the distal end of the guide tube, its devicecontroller drives its device to its retraction configuration.

As an alternative to the method described in reference to 3803-3804,rather than waiting until the most proximal rotated link of each devicereaches a threshold distance “TH” from the distal end of the guide tubebefore the device controller starts driving the device to its retractionconfiguration, each of the device controllers may start driving itsdevice to the retraction configuration immediately upon receiving acommand indicating desired movement in the retraction direction. In thiscase, each device controller is configured to drive its device to itsretraction configuration in a manner that ensures that any rotated linkof the device is properly aligned to freely enter the device's channelprior to its entry into the channel while avoiding harm to the patientand collisions with other devices.

While driving the imaging system to its retraction configuration, it isimportant to keep in mind that the imaging system controller uses thereceived information of the position of the associated instrument's endeffector to command movement of its image capturing end to maintain theend effector in its field of view. Since the operator is viewing theimage captured by the image capturing end on a display screen whilemoving the input device, the operator may become disoriented and/or movethe input device in an incorrect manner to properly command retractionof its associated instrument. To compensate for such a non-intuitiveexperience, the reference frames (i.e. blocks 303 and 312 of FIG. 23)used to compute kinematics of the master and slave manipulators (i.e.,blocks 302 and 311 of FIG. 23) are modified such that theposition/orientation of the master with respect to the display screenbeing viewed by the operator constantly corresponds to the position andorientation of the tip (e.g., point on the end effector) of theassociated instrument with respect to the tip (e.g., point on the imagecapturing end) of the imaging system.

In the event that a device controller subsequently receives an insertioncommand (i.e., a command to move the device in a direction extendingaway from the distal end of the guide tube), the device controller mayautomatically drive the device to a desired operational configuration.The desired operational configuration may be a preferred configurationstored in a memory device associated with one or more processors thatimplement the various controllers and processes described herein.Alternatively, it may be a previously assumed operational configurationthat has been stored the memory device. As an example of this lattercase, the device joint positions for the operational configurations ofthe devices just prior to initiating their retraction towards the guidetube may be stored in the memory device so that if the Surgeon decidesto re-insert the devices (or their replacement devices after a toolexchange procedure), their device controllers may automatically drivethe devices back to the stored operational configurations.

In some instances a surgical instrument is removable and may be replacedwith a different surgical instrument that has a structure similar toinstrument but a different end effector so as to perform a differentsurgical task. Accordingly, a single guide tube may be used for one ormore interchangeable surgical instruments. In one instance the endeffector of the surgical instrument is removable so that it may bereadily exchanged with another. In another instance, a surgicalaccessory such as a clip or suturing material may be provided to agrasping end effector for delivery to the work site while guide tuberemains in the patient. A convenient way of performing such end effectorexchange (also referred to herein as a “tool exchange”) or providingsuch a surgical accessory to a retracted grasping end effector is to usea fenestrated guide tube wherein one or more cut-outs are provided inthe guide tube in a part externally extending out of the patient whileanother part of the guide tube extends internally into the patientthrough the entry aperture.

FIG. 39 illustrates an example of using coupled control for retractingmedical devices into a fenestrated guide tube for a tool exchange orother purpose such as delivering a surgical accessory to the work site.In the example, a plurality of devices including an imaging system andat least two instruments extend through and beyond the distal end of aguide tube.

In 3901, a retraction command is received from an input deviceassociated with an instrument to be retracted (referred to herein as the“associated instrument”). The retraction command is indicated bymovement of the input device in a direction that would result incommanding the associated instrument to be retracted back towards and/orinto a distal end of the guide tube. As previously described inreference to FIGS. 23 and 32, sensed joint movement of the input deviceis processed by the instrument's master/slave control system and theresulting commanded state of the distal tip of the associated instrumentis picked off at the output of the scale & offset block and providedalong with information identifying the associated instrument (and inparticular, its placement in the guide tube) through a coupling block tocoupled controller blocks of other devices in the system.

In 3902, a determination is made within each of the coupled controllerblocks whether its associated device is to be retracted back along withthe associated instrument. In the case of the coupled controller blockfor the imaging system, the determination is affirmative so that theoperator may continuously view the working end of the associatedinstrument as it is retracted back into the guide tube. In the case ofthe coupled controller blocks of other devices, the determination takesinto account whether their respective instruments would be blockingaccess to the associated instrument's end effector from an opening inthe guide tube through which the tool exchange and providing of asurgical accessory is to take place. If the unretracted instrument wouldblock such access to the associated instrument's end effector throughthe opening, then the determination for coupled controller block of theblocking instrument would also be affirmative. On the other hand, thedetermination for coupled controller blocks of non-blocking instrumentswould be negative.

In 3903, coupled controller blocks making affirmative determinationsthen relay the received retraction commands to their respectivecontrollers, which in turn, command their respective device manipulatorsto retract their devices (referred to herein as the “coupled devices”)in the desired retraction direction from their positions at the time.

In 3904, each of the retracting device controllers (for both theassociated device and the coupled devices) determines when the proximalend of the most proximal rotated link of its respective device is withina threshold distance “TH” from the distal end of the guide tube in themanner described in reference to 3803 of FIG. 38.

In 3905, each of the device controllers then commands its respectivedevice manipulator to drive its device to a retraction configuration(i.e., a joint and link configuration that allows the device to be fullyretracted into the guide tube) upon determining that the proximal end ofthe most proximal rotated link of its respective device is within thethreshold distance “TH” from the distal end of the guide tube in themanner described in reference to 3804 of FIG. 38 (including compensatingfor the moving imaging system as described therein).

In 3906, once each device being retracted is determined by itsrespective device controller to be in its retraction configuration, thedevice controller allows its respective device to be retracted into itschannel in the guide tube in response to retraction commands issued fromthe input device.

In 3907, once the operator determines that the end effector of theassociated instrument is in proper position relative to the opening inthe fenestrated guide tube, movement of the input device andconsequently, the associated instrument is stopped. The imaging system,however, may continue to move to ensure that the associated instrument'send effector is properly within its field of view. In addition, theblocking instrument continues to move until it no longer blocks theaccess to the associated instrument's end effector through the openingin the fenestrated guide tube. After access to the associatedinstrument's end effector is clear from the opening, then an exchange ofend effectors may be performed and/or a surgical accessory may beprovided to the end effector as the imaging system views the activitywith the associated instrument's end effector.

FIG. 40 illustrates an example of using coupled control for extendingmedical devices out of the guide tube and inserting it towards a worksite. Although any one of the devices may be directly controlled whilethe others are indirectly controlled for insertion towards the worksite, the example assumes the imaging system is being directlycontrolled for insertion while the instruments are indirectly controlledthrough coupled control to follow the image capturing end of the imagingsystem. Using the imaging system to lead the insertion is advantageoussince it allows the surgeon to see the path towards the work site.

Control during insertion may be accomplished, for example, in a mannersimilar to telemanipulated endoscope control in the da Vinci® SurgicalSystem—in one aspect the surgeon virtually moves the image with one orboth of the masters; she uses the masters to move the image side to sideand to pull it towards herself, consequently commanding the imagingsystem and its associated instrument assembly (e.g., a flexible guidetube) to steer towards a fixed center point on the output display and toadvance inside the patient. In one aspect the camera control is designedto give the impression that the masters are fixed to the image so thatthe image moves in the same direction that the master handles are moved,as in the da Vinci® surgical system. This design causes the masters tobe in the correct location to control the instruments when the surgeonexits from camera control, and consequently it avoids the need to clutch(disengage), move, and declutch (engage) the masters back into positionprior to beginning or resuming instrument control. In some aspects themaster position may be made proportional to the insertion velocity toavoid using a large master workspace. Alternatively, the surgeon mayclutch and declutch the masters to use a ratcheting action forinsertion. In some aspects, insertion (e.g., past the glottis whenentering via the esophagus) may be controlled manually (e.g., by handoperated wheels), and automated insertion (e.g., servomotor drivenrollers) is then done when the distal end of the surgical instrumentassembly is near the surgical site. Preoperative or real time image data(e.g., MRI, X-ray) of the patient's anatomical structures and spacesavailable for insertion trajectories may be used to assist insertion.

In 4001, the imaging system controller receives an insertion commandfrom an associated input device. In 4002, the imaging system controllercommands the imaging system manipulator to move the imaging system inresponse to insertion command, while in 4003, the imaging systemcontroller provides the movement command to other device coupledcontrollers so that they may also command their respective devices tomove in response to the imaging system commanded movement. In 4004, theimaging system controller determines whether the image capturing end ofthe imaging system has reached its desired position. This determinationmay be performed either automatically based upon programmed criteria orit may be indicated through action taken by the Surgeon such asdepressing a button on the input device associated with the imagingsystem at the time. In 4005, after the imaging system controller hasdetermined that the image capturing end of the imaging system hasreached its desired position it provides an indication of such to theinstrument coupled controllers (e.g., the motion controller 2202 or theinstrument optimizer/controllers 2224 a,2224 b,2224 c, depending uponwhich instruments are to be deployed and whether a centralized ordistributed coupled mode architecture is employed) so that theinstrument controllers in response thereof command their respectiveinstrument manipulators to move their instruments into their optimaloperating configurations. Placing the devices in their optimal operatingconfigurations in this case generally involve placing the working endsof the instruments within the field of view of the imaging system andoptimizing the workspaces of the instruments (such as shown, forexample, in FIG. 18).

As apparent from the coupled control mode examples described herein, notall position information provided to the motion coordinator or thedevice optimizer/controllers is used. Therefore, either more informationthan is necessary is transmitted between the device controllers withsome of it being ignored or only necessary information is transmitted.Although the descriptions of FIGS. 30-31 may indicate the former, it isto be appreciated that the implementations described therein may alsoapply to the latter.

It is further noted that any time the image capturing end of the imagingsystem moves as a coupled device, the image reference frame used by theSurgeon for master/slave teleoperation changes and such change mayaffect the ability of Surgeon to perform precise surgical motions. Insuch case, a number of actions may be taken for large motions of theimaging capturing end of the imaging system. For example, hapticfeedback may be provided on the input device to assist the Surgeon totake appropriate action, or a computer generated auxiliary view ofdevices extending out of a distal end of a guide tube may be providedfrom a stable (e.g., fixed) perspective and relied upon by the Surgeonfor master/slave teleoperation, or the images captured by the imagingsystem may be modified in real-time to maintain an intuitively correctmaster/slave mapping with the modified images displayed on the surgeonconsole.

These descriptions of examples of various minimally invasive surgicalsystems, assemblies, and instruments, and of the associated components,are not to be taken as limiting. It should be understood that manyvariations that incorporate the aspects described herein are possible.For example, various combinations of rigid and flexible instruments andinstrument components, and of guide tubes and guide tube components,fall within the scope of this description. The claims define theinvention.

What is claimed is:
 1. A medical system comprising: a mastermanipulator; a first slave manipulator coupleable to a first device; asecond slave manipulator coupleable to a second device; a first slavecontroller operatively coupleable to the master manipulator and thefirst slave manipulator so that the first slave controller is configuredto control the first slave manipulator for moving the first device to afirst device commanded position indicated by an input from the mastermanipulator; and a second slave controller operatively coupleable to thefirst slave controller and the second slave manipulator so that thesecond slave controller is configured to control the second slavemanipulator for moving the second device, to perform a secondaryobjective, according to information of the first device commandedposition received from the first slave controller.
 2. The medical systemaccording to claim 1, wherein the master manipulator has a plurality ofmaster joints; and wherein the first slave controller is configured todetermine the first device commanded position from information of masterjoint positions of the plurality of master joints.
 3. The medical systemaccording to claim 2, wherein the first slave manipulator has aplurality of first slave joints coupled to a plurality of first linksfor moving the first device when the first device is coupled to thefirst slave manipulator; and wherein the first slave controller isconfigured to: receive the information of the master joint positionsover time from a plurality of master joint position sensors coupled tothe master joints, transform the information of the master jointpositions over time to the first device commanded position, generatefirst slave joint commanded positions by using the first devicecommanded position, and control movement of the plurality of first slavejoints according to the first slave joint commanded positions.
 4. Themedical system according to claim 3, wherein the second slave controlleris coupled to the first slave controller via a data pick-off/receivingpoint.
 5. The medical system according to claim 1, wherein the firstdevice comprises a first medical tool; wherein the second devicecomprises a base; wherein the base is coupleable to the first medicaltool so that the first medical tool moves when the base moves; andwherein the secondary objective is to move the base so that a range ofmotion of the first device is increased.
 6. The medical system accordingto claim 5, wherein the base comprises a guide tube; and wherein thefirst medical tool is extendable through, and out of a distal end of,the guide tube.
 7. The medical system according to claim 1, wherein thefirst device comprises an imaging system; wherein the second devicecomprises a second medical tool; and wherein the secondary objective isto move the second medical tool so that a working end of the secondmedical tool is within a field of view of the imaging system.
 8. Themedical system according to claim 1, wherein the first device comprisesa first medical tool; wherein the second device comprises an imagingsystem; and wherein the secondary objective is to move the imagingsystem so that a working end of the first medical tool is within a fieldof view of the imaging system.
 9. The medical system according to claim1, wherein the first device comprises a guide tube; wherein the seconddevice comprises a second medical tool, the second medical tool beingextendable through, and out of a distal end of, the guide tube; whereinthe first slave controller is configured to control the first slavemanipulator for moving the guide tube in one or more directions asindicated by the input from the master manipulator, but not in a firstdirection parallel to a longitudinal axis of the guide tube; and whereinthe second slave controller is configured to control the second slavemanipulator for moving the second medical tool in a second direction,the second direction being parallel to the longitudinal axis of theguide tube when the input from the master manipulator indicates movingthe guide tube in the first direction.
 10. The medical system accordingto claim 1, further comprising: a third slave manipulator coupleable toa third device; and a third slave controller operatively coupleable tothe first slave controller and the third slave manipulator so that thethird slave controller is configured to control the third slavemanipulator for moving the third device, to perform the secondaryobjective in conjunction with the second slave controller, according toinformation of the first device commanded position received from thefirst slave controller.
 11. A medical system comprising: a mastermanipulator; first manipulator means for moving a first device portion;second manipulator means for moving a second device portion; firstcontroller means for controlling the first manipulator means to move thefirst device portion to a commanded position indicated by an input fromthe master manipulator; and second controller means for controlling thesecond manipulator means to move the second device portion according toinformation of the commanded position received from the first controllermeans.
 12. The medical system according to claim 11, wherein the secondcontroller means controls the second manipulator means to move thesecond device portion according to information of the commanded positionreceived from the first controller means to perform a secondaryobjective.
 13. The medical system according to claim 12, wherein thefirst device portion comprises a part of a first medical tool; whereinthe second device portion comprises a part of a base; wherein the baseis coupled to the first manipulator means so that the first manipulatormeans moves when the base moves; and wherein the secondary objective isto move the base so that a range of motion of the first medical tool isincreased.
 14. The medical system according to claim 12, wherein thefirst device portion comprises a part of an imaging system; wherein thesecond device portion comprises a part of a second medical tool; andwherein the secondary objective is to move the second medical tool sothat a working end of the second medical tool is within a field of viewof the imaging system.
 15. The medical system according to claim 12,wherein the first device portion comprises a part of a first medicaltool; wherein the second device portion comprises a part of an imagingsystem; and wherein the secondary objective is to move the imagingsystem so that a working end of the first medical tool is within a fieldof view of the imaging system.
 16. The medical system according to claim11, further comprising: third manipulator means for moving a thirddevice portion; and third controller means for controlling the thirdmanipulator means to move the third device portion to perform asecondary objective in conjunction with the second controller meansaccording to information of the commanded position received from thefirst controller means.
 17. A method comprising: moving a first deviceportion according to commanded movement received from a mastermanipulator; and moving a second device portion to perform a secondaryobjective according to information of the commanded movement.
 18. Themethod of claim 17, wherein the first device portion comprises a part ofa medical tool; wherein the second device portion comprises a part of abase; wherein the base is coupled to the medical tool so that themedical tool moves when the base moves; and wherein the secondaryobjective is to move the base so that a range of motion of the medicaltool is increased.
 19. The method of claim 17, wherein the first deviceportion comprises a part of an imaging system; wherein the second deviceportion comprises a part of a medical tool; and wherein the secondaryobjective is to move the part of the medical tool so that a working endof the medical tool is within a field of view of the imaging system. 20.The method of claim 17, wherein the first device portion comprises apart of a medical tool; wherein the second device portion comprises apart of an imaging system; and wherein the secondary objective is tomove the part of the imaging system so that a working end of the medicaltool is within a field of view of the imaging system.